Oct 30 03:55:18.056788 ip-10-0-138-160 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 03:55:18.056799 ip-10-0-138-160 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 03:55:18.056806 ip-10-0-138-160 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 03:55:18.057026 ip-10-0-138-160 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 03:55:28.081648 ip-10-0-138-160 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 03:55:28.081668 ip-10-0-138-160 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot cc0f588970a343e899cace2918b0c4d7 --
Oct 30 03:57:20.261539 ip-10-0-138-160 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 03:57:20.664063 ip-10-0-138-160 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 03:57:20.664063 ip-10-0-138-160 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 03:57:20.664063 ip-10-0-138-160 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 03:57:20.664063 ip-10-0-138-160 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 03:57:20.667613 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667532    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:57:20.667613 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667612    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:57:20.667613 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667616    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:57:20.667705 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667620    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:57:20.667705 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667623    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:57:20.667705 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667626    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:57:20.667705 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667630    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:57:20.667705 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667632    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:57:20.667705 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667635    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:57:20.667705 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667638    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:57:20.667705 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667641    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:57:20.667705 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667647    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:57:20.667705 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667650    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:57:20.667705 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667653    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:57:20.667705 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667655    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:57:20.667705 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667658    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:57:20.667705 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667661    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:57:20.667705 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667663    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:57:20.667705 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667666    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:57:20.667705 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667669    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:57:20.667705 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667671    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:57:20.667705 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667674    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:57:20.667705 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667676    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:57:20.668203 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667679    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:57:20.668203 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667682    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:57:20.668203 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667685    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:57:20.668203 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667688    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:57:20.668203 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667691    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:57:20.668203 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667694    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:57:20.668203 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667696    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:57:20.668203 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667699    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:57:20.668203 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667701    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:57:20.668203 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667704    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:57:20.668203 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667706    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:57:20.668203 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667709    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:57:20.668203 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667712    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:57:20.668203 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667715    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:57:20.668203 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667718    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:57:20.668203 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667721    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:57:20.668203 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667723    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:57:20.668203 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667726    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:57:20.668203 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667729    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:57:20.668203 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667731    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:57:20.668698 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667734    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:57:20.668698 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667737    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:57:20.668698 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667739    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:57:20.668698 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667742    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:57:20.668698 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667744    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:57:20.668698 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667746    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:57:20.668698 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667749    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:57:20.668698 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667752    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:57:20.668698 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667754    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:57:20.668698 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667756    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:57:20.668698 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667759    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:57:20.668698 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667761    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:57:20.668698 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667764    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:57:20.668698 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667768    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:57:20.668698 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667770    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:57:20.668698 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667773    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:57:20.668698 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667777    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:57:20.668698 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667782    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:57:20.668698 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667786    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:57:20.669163 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667789    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:57:20.669163 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667792    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:57:20.669163 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667795    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:57:20.669163 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667798    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:57:20.669163 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667801    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:57:20.669163 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667804    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:57:20.669163 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667807    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:57:20.669163 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667809    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:57:20.669163 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667812    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:57:20.669163 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667816    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:57:20.669163 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667820    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:57:20.669163 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667823    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:57:20.669163 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667826    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:57:20.669163 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667829    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:57:20.669163 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667832    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:57:20.669163 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667835    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:57:20.669163 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667838    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:57:20.669163 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667840    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:57:20.669163 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667843    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:57:20.669641 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667846    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:57:20.669641 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667848    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:57:20.669641 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667851    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:57:20.669641 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667854    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:57:20.669641 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667856    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:57:20.669641 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667859    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:57:20.669641 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667862    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:57:20.669641 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667866    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:57:20.669641 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667868    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:57:20.669641 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667871    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:57:20.669641 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667874    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:57:20.669641 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667877    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:57:20.669641 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667879    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:57:20.669641 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667882    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:57:20.669641 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667885    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:57:20.669641 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667887    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:57:20.669641 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667890    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:57:20.669641 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667893    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:57:20.669641 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667895    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:57:20.670084 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667898    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:57:20.670084 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667901    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:57:20.670084 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667903    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:57:20.670084 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667906    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:57:20.670084 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667909    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:57:20.670084 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.667912    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:57:20.670084 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668724    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:57:20.670084 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668732    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:57:20.670084 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668735    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:57:20.670084 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668738    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:57:20.670084 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668740    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:57:20.670084 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668743    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:57:20.670084 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668746    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:57:20.670084 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668749    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:57:20.670084 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668752    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:57:20.670084 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668754    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:57:20.670084 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668757    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:57:20.670084 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668760    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:57:20.670084 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668763    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:57:20.670630 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668767    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:57:20.670630 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668770    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:57:20.670630 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668773    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:57:20.670630 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668776    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:57:20.670630 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668778    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:57:20.670630 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668781    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:57:20.670630 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668784    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:57:20.670630 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668786    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:57:20.670630 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668789    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:57:20.670630 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668792    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:57:20.670630 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668794    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:57:20.670630 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668797    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:57:20.670630 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668799    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:57:20.670630 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668802    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:57:20.670630 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668805    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:57:20.670630 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668807    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:57:20.670630 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668809    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:57:20.670630 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668813    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:57:20.670630 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668817    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:57:20.671083 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668823    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:57:20.671083 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668826    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:57:20.671083 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668828    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:57:20.671083 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668831    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:57:20.671083 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668833    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:57:20.671083 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668836    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:57:20.671083 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668838    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:57:20.671083 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668841    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:57:20.671083 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668843    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:57:20.671083 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668846    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:57:20.671083 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668848    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:57:20.671083 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668851    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:57:20.671083 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668853    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:57:20.671083 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668856    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:57:20.671083 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668859    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:57:20.671083 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668862    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:57:20.671083 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668864    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:57:20.671083 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668867    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:57:20.671083 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668870    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:57:20.671083 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668872    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:57:20.671560 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668875    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:57:20.671560 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668878    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:57:20.671560 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668880    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:57:20.671560 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668882    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:57:20.671560 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668885    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:57:20.671560 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668887    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:57:20.671560 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668890    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:57:20.671560 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668893    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:57:20.671560 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668895    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:57:20.671560 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668897    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:57:20.671560 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668900    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:57:20.671560 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668902    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:57:20.671560 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668905    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:57:20.671560 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668908    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:57:20.671560 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668911    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:57:20.671560 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668913    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:57:20.671560 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668915    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:57:20.671560 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668918    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:57:20.671560 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668921    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:57:20.671560 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668924    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:57:20.672063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668926    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:57:20.672063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668929    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:57:20.672063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668931    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:57:20.672063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668934    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:57:20.672063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668938    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:57:20.672063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668942    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:57:20.672063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668945    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:57:20.672063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668948    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:57:20.672063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668950    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:57:20.672063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668953    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:57:20.672063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668956    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:57:20.672063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668958    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:57:20.672063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668961    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:57:20.672063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668963    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:57:20.672063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668966    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:57:20.672063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668968    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:57:20.672063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668971    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:57:20.672063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668973    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:57:20.672063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668976    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:57:20.672527 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668979    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:57:20.672527 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668981    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:57:20.672527 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668984    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:57:20.672527 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668986    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:57:20.672527 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668989    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:57:20.672527 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668991    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:57:20.672527 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668994    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:57:20.672527 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.668997    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:57:20.672527 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669001    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:57:20.672527 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669003    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:57:20.672527 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669006    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:57:20.672527 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669008    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:57:20.672527 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669011    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:57:20.672527 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669013    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:57:20.672527 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669016    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:57:20.672527 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669507    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:57:20.672527 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669513    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:57:20.672527 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669516    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:57:20.672527 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669519    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:57:20.672967 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669522    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:57:20.672967 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669524    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:57:20.672967 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669527    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:57:20.672967 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669530    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:57:20.672967 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669533    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:57:20.672967 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669536    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:57:20.672967 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669538    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:57:20.672967 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669541    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:57:20.672967 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669544    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:57:20.672967 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669547    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:57:20.672967 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669550    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:57:20.672967 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669553    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:57:20.672967 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669556    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:57:20.672967 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669559    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:57:20.672967 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669562    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:57:20.672967 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669564    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:57:20.672967 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669568    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:57:20.672967 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669573    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:57:20.672967 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669576    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:57:20.672967 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669579    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:57:20.673454 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669583    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:57:20.673454 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669587    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:57:20.673454 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669590    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:57:20.673454 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669592    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:57:20.673454 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669595    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:57:20.673454 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669598    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:57:20.673454 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669600    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:57:20.673454 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669603    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:57:20.673454 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669605    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:57:20.673454 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669608    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:57:20.673454 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669610    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:57:20.673454 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669613    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:57:20.673454 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669615    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:57:20.673454 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669618    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:57:20.673454 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669620    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:57:20.673454 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669623    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:57:20.673454 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669626    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:57:20.673454 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669628    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:57:20.673454 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669631    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:57:20.673454 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669634    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:57:20.673932 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669636    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:57:20.673932 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669639    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:57:20.673932 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669641    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:57:20.673932 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669644    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:57:20.673932 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669646    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:57:20.673932 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669650    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:57:20.673932 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669654    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:57:20.673932 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669657    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:57:20.673932 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669660    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:57:20.673932 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669663    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:57:20.673932 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669666    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:57:20.673932 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669669    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:57:20.673932 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669671    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:57:20.673932 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669675    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:57:20.673932 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669678    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:57:20.673932 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669681    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:57:20.673932 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669683    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:57:20.673932 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669686    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:57:20.673932 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669688    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:57:20.674384 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669690    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:57:20.674384 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669693    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:57:20.674384 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669695    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:57:20.674384 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669698    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:57:20.674384 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669700    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:57:20.674384 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669703    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:57:20.674384 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669705    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:57:20.674384 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669708    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:57:20.674384 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669710    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:57:20.674384 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669712    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:57:20.674384 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669715    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:57:20.674384 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669717    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:57:20.674384 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669720    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:57:20.674384 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669722    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:57:20.674384 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669725    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:57:20.674384 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669728    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:57:20.674384 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669730    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:57:20.674384 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669733    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:57:20.674384 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669736    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:57:20.674827 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669739    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:57:20.674827 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669741    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:57:20.674827 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669744    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:57:20.674827 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669746    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:57:20.674827 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669749    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:57:20.674827 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669751    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:57:20.674827 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669754    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:57:20.674827 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669757    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:57:20.674827 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669761    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:57:20.674827 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669763    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:57:20.674827 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669765    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:57:20.674827 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669768    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:57:20.674827 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669771    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:57:20.674827 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669773    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:57:20.674827 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669776    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:57:20.674827 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669778    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:57:20.674827 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669781    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:57:20.674827 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669783    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:57:20.674827 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669786    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:57:20.674827 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669788    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:57:20.675301 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669791    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:57:20.675301 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669793    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:57:20.675301 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669798    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:57:20.675301 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.669801    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:57:20.675301 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670884    2578 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 03:57:20.675301 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670894    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 03:57:20.675301 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670900    2578 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 03:57:20.675301 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670906    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 03:57:20.675301 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670910    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 03:57:20.675301 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670914    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 03:57:20.675301 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670918    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 03:57:20.675301 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670923    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 03:57:20.675301 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670926    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 03:57:20.675301 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670929    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 03:57:20.675301 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670933    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 03:57:20.675301 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670937    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 03:57:20.675301 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670940    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 03:57:20.675301 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670943    2578 flags.go:64] FLAG: --cgroup-root=""
Oct 30 03:57:20.675301 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670946    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 03:57:20.675301 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670949    2578 flags.go:64] FLAG: --client-ca-file=""
Oct 30 03:57:20.675301 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670952    2578 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 03:57:20.675301 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670955    2578 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 03:57:20.675301 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670960    2578 flags.go:64] FLAG: --cluster-domain=""
Oct 30 03:57:20.675832 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670963    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 03:57:20.675832 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670966    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 03:57:20.675832 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670969    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 03:57:20.675832 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670973    2578 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 03:57:20.675832 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670976    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 03:57:20.675832 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670979    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 03:57:20.675832 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670982    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 03:57:20.675832 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670986    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 03:57:20.675832 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670989    2578 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 03:57:20.675832 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670992    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 03:57:20.675832 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.670995    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 03:57:20.675832 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671000    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 03:57:20.675832 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671003    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 03:57:20.675832 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671007    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 03:57:20.675832 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671011    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 03:57:20.675832 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671014    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 03:57:20.675832 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671017    2578 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 03:57:20.675832 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671020    2578 flags.go:64] FLAG: --enable-server="true"
Oct 30 03:57:20.675832 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671022    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 03:57:20.675832 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671026    2578 flags.go:64] FLAG: --event-burst="100"
Oct 30 03:57:20.675832 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671030    2578 flags.go:64] FLAG: --event-qps="50"
Oct 30 03:57:20.675832 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671033    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 03:57:20.675832 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671036    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 03:57:20.675832 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671040    2578 flags.go:64] FLAG: --eviction-hard=""
Oct 30 03:57:20.676453 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671044    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 03:57:20.676453 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671047    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 03:57:20.676453 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671050    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 03:57:20.676453 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671053    2578 flags.go:64] FLAG: --eviction-soft=""
Oct 30 03:57:20.676453 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671056    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 03:57:20.676453 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671059    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 03:57:20.676453 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671062    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 03:57:20.676453 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671066    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 03:57:20.676453 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671069    2578 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 03:57:20.676453 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671072    2578 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 03:57:20.676453 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671074    2578 flags.go:64] FLAG: --feature-gates=""
Oct 30 03:57:20.676453 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671079    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 03:57:20.676453 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671082    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 03:57:20.676453 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671085    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 03:57:20.676453 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671088    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 03:57:20.676453 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671091    2578 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 03:57:20.676453 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671095    2578 flags.go:64] FLAG: --help="false"
Oct 30 03:57:20.676453 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671098    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-138-160.ec2.internal"
Oct 30 03:57:20.676453 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671101    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 03:57:20.676453 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671105    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 03:57:20.676453 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671108    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 03:57:20.676453 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671111    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 03:57:20.676453 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671115    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 03:57:20.677004 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671118    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 03:57:20.677004 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671121    2578 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 03:57:20.677004 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671123    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 03:57:20.677004 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671126    2578 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 03:57:20.677004 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671129    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 03:57:20.677004 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671132    2578 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 03:57:20.677004 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671135    2578 flags.go:64] FLAG: --kube-reserved=""
Oct 30 03:57:20.677004 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671140    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 03:57:20.677004 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671142    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 03:57:20.677004 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671146    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 03:57:20.677004 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671149    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 03:57:20.677004 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671152    2578 flags.go:64] FLAG: --lock-file=""
Oct 30 03:57:20.677004 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671155    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 03:57:20.677004 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671158    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 03:57:20.677004 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671161    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 03:57:20.677004 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671166    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 03:57:20.677004 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671169    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 03:57:20.677004 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671173    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 03:57:20.677004 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671176    2578 flags.go:64] FLAG: --logging-format="text"
Oct 30 03:57:20.677004 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671178    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 03:57:20.677004 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671182    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 03:57:20.677004 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671184    2578 flags.go:64] FLAG: --manifest-url=""
Oct 30 03:57:20.677004 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671187    2578 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 03:57:20.677004 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671191    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 03:57:20.677004 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671195    2578 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671198    2578 flags.go:64] FLAG: --max-pods="110"
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671201    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671204    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671207    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671210    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671213    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671216    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671219    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671226    2578 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671229    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671232    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671247    2578 flags.go:64] FLAG: --pod-cidr=""
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671250    2578 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671253    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671258    2578 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671261    2578 flags.go:64] FLAG: --port="10250"
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671264    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671267    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05d99f2bdb0bdb8d8"
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671270    2578 flags.go:64] FLAG: --qos-reserved=""
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671273    2578 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671276    2578 flags.go:64] FLAG: --register-node="true"
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671279    2578 flags.go:64] FLAG: --register-with-taints=""
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671283    2578 flags.go:64] FLAG: --registry-burst="10"
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671285    2578 flags.go:64] FLAG: --registry-qps="5"
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671288    2578 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 03:57:20.677618 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671291    2578 flags.go:64] FLAG: --reserved-memory=""
Oct 30 03:57:20.678639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671295    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 03:57:20.678639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671298    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 03:57:20.678639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671301    2578 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 03:57:20.678639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671304    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 03:57:20.678639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671307    2578 flags.go:64] FLAG: --runonce="false"
Oct 30 03:57:20.678639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671310    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 03:57:20.678639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671313    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 03:57:20.678639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671316    2578 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 03:57:20.678639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671319    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 03:57:20.678639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671322    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 03:57:20.678639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671325    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 03:57:20.678639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671328    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 03:57:20.678639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671331    2578 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 03:57:20.678639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671334    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 03:57:20.678639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671337    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 03:57:20.678639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671340    2578 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 03:57:20.678639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671343    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 03:57:20.678639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671346    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 03:57:20.678639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671348    2578 flags.go:64] FLAG: --system-cgroups=""
Oct 30 03:57:20.678639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671351    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 03:57:20.678639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671358    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 03:57:20.678639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671361    2578 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 03:57:20.678639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671364    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 03:57:20.678639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671368    2578 flags.go:64] FLAG: --tls-min-version=""
Oct 30 03:57:20.678639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671370    2578 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 03:57:20.679701 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671373    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 03:57:20.679701 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671376    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 03:57:20.679701 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671379    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 03:57:20.679701 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671382    2578 flags.go:64] FLAG: --v="2"
Oct 30 03:57:20.679701 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671386    2578 flags.go:64] FLAG: --version="false"
Oct 30 03:57:20.679701 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671390    2578 flags.go:64] FLAG: --vmodule=""
Oct 30 03:57:20.679701 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671394    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 03:57:20.679701 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671398    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 03:57:20.679701 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671556    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:57:20.679701 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671560    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:57:20.679701 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671563    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:57:20.679701 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671566    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:57:20.679701 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671569    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:57:20.679701 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671572    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:57:20.679701 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671575    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:57:20.679701 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671578    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:57:20.679701 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671580    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:57:20.679701 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671583    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:57:20.679701 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671586    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:57:20.679701 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671589    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:57:20.679701 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671591    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:57:20.680477 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671594    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:57:20.680477 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671597    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:57:20.680477 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671599    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:57:20.680477 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671603    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:57:20.680477 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671607    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:57:20.680477 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671610    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:57:20.680477 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671615    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:57:20.680477 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671618    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:57:20.680477 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671621    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:57:20.680477 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671623    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:57:20.680477 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671626    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:57:20.680477 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671628    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:57:20.680477 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671631    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:57:20.680477 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671633    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:57:20.680477 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671636    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:57:20.680477 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671639    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:57:20.680477 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671641    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:57:20.680477 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671644    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:57:20.680477 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671646    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:57:20.680931 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671649    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:57:20.680931 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671652    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:57:20.680931 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671654    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:57:20.680931 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671657    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:57:20.680931 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671659    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:57:20.680931 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671662    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:57:20.680931 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671664    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:57:20.680931 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671668    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:57:20.680931 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671671    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:57:20.680931 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671674    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:57:20.680931 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671676    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:57:20.680931 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671679    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:57:20.680931 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671682    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:57:20.680931 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671684    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:57:20.680931 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671687    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:57:20.680931 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671690    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:57:20.680931 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671692    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:57:20.680931 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671695    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:57:20.680931 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671698    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:57:20.680931 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671702    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:57:20.681816 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671705    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:57:20.681816 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671708    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:57:20.681816 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671710    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:57:20.681816 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671713    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:57:20.681816 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671715    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:57:20.681816 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671718    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:57:20.681816 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671720    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:57:20.681816 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671723    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:57:20.681816 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671726    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:57:20.681816 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671728    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:57:20.681816 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671731    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:57:20.681816 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671733    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:57:20.681816 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671736    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:57:20.681816 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671739    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:57:20.681816 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671742    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:57:20.681816 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671744    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:57:20.681816 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671747    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:57:20.681816 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671749    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:57:20.681816 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671752    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:57:20.682689 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671755    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:57:20.682689 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671758    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:57:20.682689 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671761    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:57:20.682689 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671763    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:57:20.682689 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671766    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:57:20.682689 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671770    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:57:20.682689 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671774    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:57:20.682689 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671777    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:57:20.682689 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671780    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:57:20.682689 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671783    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:57:20.682689 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671785    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:57:20.682689 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671788    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:57:20.682689 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671792    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:57:20.682689 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671794    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:57:20.682689 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671797    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:57:20.682689 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671800    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:57:20.682689 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671802    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:57:20.682689 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671805    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:57:20.682689 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671807    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:57:20.682689 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671810    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:57:20.683483 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671813    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:57:20.683483 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671815    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:57:20.683483 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671818    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:57:20.683483 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671820    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:57:20.683483 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671823    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:57:20.683483 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671825    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:57:20.683483 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671828    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:57:20.683483 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671831    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:57:20.683483 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671834    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:57:20.683483 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671836    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:57:20.683483 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671839    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:57:20.683483 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671841    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:57:20.683483 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671845    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:57:20.683483 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671848    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:57:20.683483 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671850    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:57:20.683483 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.671876    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 03:57:20.683903 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671910    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:57:20.683903 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671915    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:57:20.683903 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671919    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:57:20.683903 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671922    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:57:20.683903 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671925    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:57:20.683903 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671928    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:57:20.683903 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671930    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:57:20.683903 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671933    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:57:20.683903 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671937    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:57:20.683903 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671940    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:57:20.683903 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671943    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:57:20.683903 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671946    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:57:20.683903 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671948    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:57:20.683903 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671951    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:57:20.683903 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671954    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:57:20.683903 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671956    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:57:20.683903 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671959    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:57:20.683903 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671961    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:57:20.683903 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671964    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:57:20.684685 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671966    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:57:20.684685 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671969    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:57:20.684685 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671972    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:57:20.684685 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671974    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:57:20.684685 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671977    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:57:20.684685 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671980    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:57:20.684685 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671983    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:57:20.684685 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671985    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:57:20.684685 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671988    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:57:20.684685 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671991    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:57:20.684685 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671995    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:57:20.684685 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.671997    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:57:20.684685 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672000    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:57:20.684685 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672003    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:57:20.684685 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672005    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:57:20.684685 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672008    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:57:20.684685 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672011    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:57:20.684685 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672013    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:57:20.684685 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672016    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:57:20.684685 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672019    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:57:20.685458 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672021    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:57:20.685458 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672026    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:57:20.685458 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672028    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:57:20.685458 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672031    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:57:20.685458 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672033    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:57:20.685458 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672036    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:57:20.685458 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672038    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:57:20.685458 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672041    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:57:20.685458 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672045    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:57:20.685458 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672048    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:57:20.685458 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672050    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:57:20.685458 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672053    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:57:20.685458 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672055    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:57:20.685458 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672058    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:57:20.685458 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672060    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:57:20.685458 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672063    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:57:20.685458 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672066    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:57:20.685458 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672069    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:57:20.685458 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672072    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:57:20.685964 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672074    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:57:20.685964 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672077    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:57:20.685964 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672080    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:57:20.685964 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672085    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:57:20.685964 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672089    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:57:20.685964 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672092    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:57:20.685964 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672094    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:57:20.685964 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672097    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:57:20.685964 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672099    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:57:20.685964 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672102    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:57:20.685964 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672105    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:57:20.685964 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672107    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:57:20.685964 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672110    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:57:20.685964 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672112    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:57:20.685964 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672116    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:57:20.685964 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672119    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:57:20.685964 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672122    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:57:20.685964 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672125    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:57:20.685964 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672127    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:57:20.685964 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672130    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:57:20.686547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672132    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:57:20.686547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672135    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:57:20.686547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672137    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:57:20.686547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672140    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:57:20.686547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672142    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:57:20.686547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672145    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:57:20.686547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672148    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:57:20.686547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672150    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:57:20.686547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672153    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:57:20.686547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672155    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:57:20.686547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672158    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:57:20.686547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672160    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:57:20.686547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672163    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:57:20.686547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672166    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:57:20.686547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672169    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:57:20.686547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672173    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:57:20.686547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672175    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:57:20.686547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672178    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:57:20.686547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672180    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:57:20.687001 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672183    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:57:20.687001 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672185    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:57:20.687001 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672188    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:57:20.687001 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672191    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:57:20.687001 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672193    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:57:20.687001 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672196    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:57:20.687001 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672199    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:57:20.687001 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672203    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:57:20.687001 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.672206    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:57:20.687001 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.680186    2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 03:57:20.687001 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.680205    2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 03:57:20.687001 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.680983    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 03:57:20.687001 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.680996    2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 03:57:20.687001 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681083    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:57:20.687001 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681091    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:57:20.687001 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681096    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:57:20.687001 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681101    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:57:20.687001 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681105    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:57:20.687001 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681109    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:57:20.687001 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681114    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:57:20.687552 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681119    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:57:20.687552 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681123    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:57:20.687552 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681127    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:57:20.687552 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681132    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:57:20.687552 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681136    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:57:20.687552 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681141    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:57:20.687552 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681145    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:57:20.687552 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681150    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:57:20.687552 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681154    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:57:20.687552 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681158    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:57:20.687552 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681162    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:57:20.687552 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681166    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:57:20.687552 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681172    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:57:20.687552 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681179    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:57:20.687552 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681186    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:57:20.687552 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681193    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:57:20.687552 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681198    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:57:20.687552 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681203    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:57:20.687552 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681207    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:57:20.688063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681212    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:57:20.688063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681216    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:57:20.688063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681220    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:57:20.688063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681227    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:57:20.688063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681232    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:57:20.688063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681252    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:57:20.688063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681257    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:57:20.688063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681262    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:57:20.688063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681266    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:57:20.688063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681270    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:57:20.688063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681275    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:57:20.688063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681279    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:57:20.688063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681282    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:57:20.688063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681287    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:57:20.688063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681291    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:57:20.688063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681296    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:57:20.688063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681300    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:57:20.688063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681305    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:57:20.688063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681309    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:57:20.688063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681314    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:57:20.688550 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681318    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:57:20.688550 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681322    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:57:20.688550 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681327    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:57:20.688550 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681331    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:57:20.688550 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681336    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:57:20.688550 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681340    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:57:20.688550 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681344    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:57:20.688550 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681348    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:57:20.688550 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681353    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:57:20.688550 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681357    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:57:20.688550 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681362    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:57:20.688550 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681366    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:57:20.688550 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681371    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:57:20.688550 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681375    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:57:20.688550 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681379    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:57:20.688550 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681384    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:57:20.688550 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681391    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:57:20.688550 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681395    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:57:20.688550 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681399    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:57:20.688550 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681404    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:57:20.689215 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681408    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:57:20.689215 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681413    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:57:20.689215 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681417    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:57:20.689215 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681421    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:57:20.689215 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681425    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:57:20.689215 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681429    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:57:20.689215 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681434    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:57:20.689215 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681438    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:57:20.689215 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681442    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:57:20.689215 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681446    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:57:20.689215 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681450    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:57:20.689215 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681455    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:57:20.689215 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681459    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:57:20.689215 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681463    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:57:20.689215 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681468    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:57:20.689215 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681472    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:57:20.689215 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681476    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:57:20.689215 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681480    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:57:20.689215 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681492    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:57:20.689955 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681496    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:57:20.689955 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681501    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:57:20.689955 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681505    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:57:20.689955 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681510    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:57:20.689955 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681514    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:57:20.689955 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681519    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:57:20.689955 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681523    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:57:20.689955 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681527    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:57:20.689955 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681532    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:57:20.689955 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681536    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:57:20.689955 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681542    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:57:20.689955 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681547    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:57:20.689955 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681552    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:57:20.689955 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681556    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:57:20.689955 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681561    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:57:20.689955 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681566    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:57:20.689955 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681570    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:57:20.689955 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681574    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:57:20.689955 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681579    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:57:20.690485 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681584    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:57:20.690485 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681588    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:57:20.690485 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.681631    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 03:57:20.690485 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681871    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:57:20.690485 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681879    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:57:20.690485 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681885    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:57:20.690485 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681889    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:57:20.690485 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681894    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:57:20.690485 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681899    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:57:20.690485 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681903    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:57:20.690485 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681907    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:57:20.690485 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681912    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:57:20.690485 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681916    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:57:20.690485 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681920    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:57:20.690485 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681925    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:57:20.691044 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681930    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:57:20.691044 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681934    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:57:20.691044 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681938    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:57:20.691044 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681943    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:57:20.691044 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681948    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:57:20.691044 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681952    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:57:20.691044 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681956    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:57:20.691044 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681960    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:57:20.691044 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681965    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:57:20.691044 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681970    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:57:20.691044 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681975    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:57:20.691044 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681979    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:57:20.691044 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681983    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:57:20.691044 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681988    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:57:20.691044 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681992    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:57:20.691044 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.681996    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:57:20.691044 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682000    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:57:20.691044 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682004    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:57:20.691044 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682009    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:57:20.691760 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682013    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:57:20.691760 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682017    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:57:20.691760 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682021    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:57:20.691760 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682025    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:57:20.691760 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682029    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:57:20.691760 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682034    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:57:20.691760 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682038    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:57:20.691760 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682044    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:57:20.691760 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682050    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:57:20.691760 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682055    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:57:20.691760 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682059    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:57:20.691760 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682064    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:57:20.691760 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682069    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:57:20.691760 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682075    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:57:20.691760 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682081    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:57:20.691760 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682088    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:57:20.691760 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682093    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:57:20.691760 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682099    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:57:20.691760 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682103    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:57:20.692547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682108    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:57:20.692547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682112    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:57:20.692547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682116    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:57:20.692547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682120    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:57:20.692547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682125    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:57:20.692547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682129    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:57:20.692547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682133    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:57:20.692547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682138    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:57:20.692547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682142    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:57:20.692547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682146    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:57:20.692547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682151    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:57:20.692547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682155    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:57:20.692547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682159    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:57:20.692547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682163    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:57:20.692547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682168    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:57:20.692547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682172    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:57:20.692547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682176    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:57:20.692547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682180    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:57:20.692547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682184    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:57:20.692547 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682188    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:57:20.693297 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682192    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:57:20.693297 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682197    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:57:20.693297 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682202    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:57:20.693297 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682206    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:57:20.693297 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682210    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:57:20.693297 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682214    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:57:20.693297 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682218    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:57:20.693297 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682222    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:57:20.693297 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682227    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:57:20.693297 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682231    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:57:20.693297 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682251    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:57:20.693297 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682256    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:57:20.693297 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682260    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:57:20.693297 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682265    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:57:20.693297 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682269    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:57:20.693297 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682273    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:57:20.693297 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682277    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:57:20.693297 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682283    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:57:20.693297 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682287    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:57:20.693297 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682291    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:57:20.694071 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682295    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:57:20.694071 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682300    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:57:20.694071 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682304    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:57:20.694071 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682308    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:57:20.694071 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682313    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:57:20.694071 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682317    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:57:20.694071 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682322    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:57:20.694071 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682326    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:57:20.694071 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682330    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:57:20.694071 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682335    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:57:20.694071 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682339    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:57:20.694071 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682343    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:57:20.694071 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682347    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:57:20.694071 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682351    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:57:20.694071 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682355    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:57:20.694071 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682360    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:57:20.694660 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.682401    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 03:57:20.694660 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682452    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:57:20.694660 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682458    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:57:20.694660 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682463    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:57:20.694660 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682467    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:57:20.694660 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682471    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:57:20.694660 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682475    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:57:20.694660 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682479    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:57:20.694660 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682483    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:57:20.694660 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682488    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:57:20.694660 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682493    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:57:20.694660 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682497    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:57:20.694660 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682502    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:57:20.694660 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682507    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:57:20.694660 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682512    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:57:20.695216 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682516    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:57:20.695216 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682520    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:57:20.695216 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682524    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:57:20.695216 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682528    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:57:20.695216 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682533    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:57:20.695216 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682537    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:57:20.695216 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682541    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:57:20.695216 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682545    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:57:20.695216 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682550    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:57:20.695216 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682554    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:57:20.695216 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682558    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:57:20.695216 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682562    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:57:20.695216 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682566    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:57:20.695216 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682571    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:57:20.695216 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682575    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:57:20.695216 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682579    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:57:20.695216 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682584    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:57:20.695216 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682588    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:57:20.695216 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682593    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:57:20.695216 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682597    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:57:20.695949 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682602    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:57:20.695949 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682606    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:57:20.695949 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682610    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:57:20.695949 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682615    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:57:20.695949 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682619    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:57:20.695949 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682623    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:57:20.695949 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682627    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:57:20.695949 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682631    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:57:20.695949 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682635    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:57:20.695949 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682639    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:57:20.695949 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682644    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:57:20.695949 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682648    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:57:20.695949 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682653    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:57:20.695949 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682657    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:57:20.695949 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682661    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:57:20.695949 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682665    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:57:20.695949 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682670    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:57:20.695949 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682674    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:57:20.695949 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682678    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:57:20.696493 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682682    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:57:20.696493 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682687    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:57:20.696493 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682691    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:57:20.696493 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682694    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:57:20.696493 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682701    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:57:20.696493 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682706    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:57:20.696493 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682711    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:57:20.696493 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682716    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:57:20.696493 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682721    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:57:20.696493 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682726    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:57:20.696493 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682733    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:57:20.696493 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682739    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:57:20.696493 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682745    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:57:20.696493 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682750    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:57:20.696493 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682754    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:57:20.696493 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682759    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:57:20.696493 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682763    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:57:20.696493 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682767    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:57:20.696493 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682771    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:57:20.696959 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682775    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:57:20.696959 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682779    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:57:20.696959 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682783    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:57:20.696959 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682788    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:57:20.696959 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682792    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:57:20.696959 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682799    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:57:20.696959 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682803    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:57:20.696959 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682810    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:57:20.696959 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682814    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:57:20.696959 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682819    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:57:20.696959 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682823    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:57:20.696959 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682827    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:57:20.696959 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682831    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:57:20.696959 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682836    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:57:20.696959 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682840    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:57:20.696959 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682844    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:57:20.696959 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682849    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:57:20.696959 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682853    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:57:20.696959 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682857    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:57:20.696959 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682861    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:57:20.697490 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682865    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:57:20.697490 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682870    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:57:20.697490 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682874    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:57:20.697490 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682879    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:57:20.697490 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682883    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:57:20.697490 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682888    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:57:20.697490 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682892    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:57:20.697490 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682896    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:57:20.697490 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682900    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:57:20.697490 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682905    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:57:20.697490 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682910    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:57:20.697490 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682915    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:57:20.697490 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682919    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:57:20.697490 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:20.682923    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:57:20.697490 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.683001    2578 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 03:57:20.697490 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.685969    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 03:57:20.697490 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.687055    2578 server.go:1014] "Starting client certificate rotation"
Oct 30 03:57:20.697490 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.687154    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 03:57:20.697490 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.687930    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 03:57:20.698001 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.689405    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 03:57:20.698001 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.697495    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 03:57:20.711520 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.711504    2578 log.go:25] "Validated CRI v1 runtime API"
Oct 30 03:57:20.715443 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.715429    2578 log.go:25] "Validated CRI v1 image API"
Oct 30 03:57:20.716214 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.716190    2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 03:57:20.719386 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.719366    2578 fs.go:135] Filesystem UUIDs: map[064bcac5-4be3-441f-8b6e-96c55953d0c2:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8da62ed4-7e90-4cf4-9157-dc8a0f041536:/dev/nvme0n1p3]
Oct 30 03:57:20.719465 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.719386    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 03:57:20.726014 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.725896    2578 manager.go:217] Machine: {Timestamp:2026-10-30 03:57:20.724407608 +0000 UTC m=+0.353927345 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3093870 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e192808e641a4239ad2a51c10792f SystemUUID:ec2e1928-08e6-41a4-239a-d2a51c10792f BootID:cc0f5889-70a3-43e8-99ca-ce2918b0c4d7 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d8:7a:70:9f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d8:7a:70:9f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:04:d3:70:e1:b2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 03:57:20.726083 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.726016    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 03:57:20.726111 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.726090    2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 03:57:20.729376 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.729356    2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 03:57:20.729524 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.729379    2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-160.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 03:57:20.729571 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.729527    2578 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 03:57:20.729571 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.729535    2578 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 03:57:20.729571 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.729566    2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 03:57:20.730417 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.730398    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 03:57:20.731588 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.731577    2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 03:57:20.735882 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.735871    2578 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 03:57:20.736002 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.735993    2578 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 03:57:20.736869 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.736859    2578 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 03:57:20.736905 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.736874    2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 03:57:20.736905 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.736885    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 03:57:20.736905 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.736894    2578 kubelet.go:405] "Adding apiserver pod source"
Oct 30 03:57:20.736905 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.736901    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 03:57:20.737323 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.737303    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 03:57:20.739821 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.739806    2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 03:57:20.747035 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.747022    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 03:57:20.747974 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.747963    2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 03:57:20.748014 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.747993    2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 03:57:20.748133 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.748123    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 03:57:20.748160 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.748138    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 03:57:20.748160 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.748144    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 03:57:20.748160 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.748149    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 03:57:20.748160 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.748154    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 03:57:20.748160 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.748160    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 03:57:20.749363 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.749337    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 03:57:20.749363 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.749362    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 03:57:20.749507 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.749376    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 03:57:20.749507 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.749386    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 03:57:20.749507 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.749403    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 03:57:20.749507 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.749419    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 03:57:20.750463 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.750367    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 03:57:20.750654 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.750643    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 03:57:20.754919 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.754903    2578 server.go:1263] "Started kubelet"
Oct 30 03:57:20.755051 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.755005    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 03:57:20.755111 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.755075    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 03:57:20.755423 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.755387    2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 03:57:20.755557 ip-10-0-138-160 systemd[1]: Started Kubernetes Kubelet.
Oct 30 03:57:20.756389 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.756373    2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 03:57:20.757360 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.757344    2578 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 03:57:20.761428 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:20.761413    2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 03:57:20.765470 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.765456    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 03:57:20.766126 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.766110    2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 03:57:20.766281 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:20.766230    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-160.ec2.internal\" not found"
Oct 30 03:57:20.766872 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.766851    2578 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 03:57:20.766872 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.766865    2578 factory.go:55] Registering systemd factory
Oct 30 03:57:20.767000 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.766879    2578 factory.go:223] Registration of the systemd container factory successfully
Oct 30 03:57:20.767000 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.766869    2578 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 03:57:20.767000 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.766854    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 03:57:20.767000 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.766950    2578 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 03:57:20.767000 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.766956    2578 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 03:57:20.767212 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.767143    2578 factory.go:153] Registering CRI-O factory
Oct 30 03:57:20.767212 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.767157    2578 factory.go:223] Registration of the crio container factory successfully
Oct 30 03:57:20.767331 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.767205    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 03:57:20.767331 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.767256    2578 factory.go:103] Registering Raw factory
Oct 30 03:57:20.767331 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.767274    2578 manager.go:1199] Started watching for new ooms in manager
Oct 30 03:57:20.768326 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.768310    2578 manager.go:319] Starting recovery of all containers
Oct 30 03:57:20.774743 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.774719    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 03:57:20.776788 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.776760    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7h4cz"
Oct 30 03:57:20.777494 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.777470    2578 manager.go:324] Recovery completed
Oct 30 03:57:20.783294 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.783278    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:57:20.784516 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.784468    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-160.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 03:57:20.786774 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.786757    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:57:20.786852 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.786791    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:57:20.786852 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.786805    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:57:20.787912 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.787896    2578 cpu_manager.go:226] "Starting" policy="none"
Oct 30 03:57:20.787912 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.787911    2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 03:57:20.788016 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.787900    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:57:20.788016 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.787930    2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 03:57:20.788580 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.788562    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7h4cz"
Oct 30 03:57:20.788648 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.788595    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:57:20.788648 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.788618    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:57:20.788648 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.788634    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:57:20.789975 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.789962    2578 policy_none.go:50] "Start"
Oct 30 03:57:20.790014 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.789979    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 03:57:20.790014 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.789989    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 03:57:20.793043 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.793029    2578 policy_none.go:44] "Start"
Oct 30 03:57:20.795033 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:20.795014    2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-160.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 03:57:20.796138 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:20.794982    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-160.ec2.internal.18e33289084eb0d1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-160.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-160.ec2.internal,},FirstTimestamp:2026-10-30 03:57:20.754872529 +0000 UTC m=+0.384392265,LastTimestamp:2026-10-30 03:57:20.754872529 +0000 UTC m=+0.384392265,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-160.ec2.internal,}"
Oct 30 03:57:20.805876 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:20.805852    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 03:57:20.805959 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:20.805924    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-160.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 03:57:20.840962 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.830004    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 03:57:20.840962 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.831107    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 03:57:20.840962 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.831129    2578 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 03:57:20.840962 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.831151    2578 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 03:57:20.840962 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:20.831192    2578 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 03:57:20.840962 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.833182    2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 03:57:20.850682 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.850666    2578 manager.go:341] "Starting Device Plugin manager"
Oct 30 03:57:20.850768 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:20.850713    2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 03:57:20.850768 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.850725    2578 server.go:87] "Starting device plugin registration server"
Oct 30 03:57:20.850970 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.850957    2578 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 03:57:20.851030 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.850971    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 03:57:20.851079 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.851032    2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 03:57:20.851301 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.851105    2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 03:57:20.851301 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.851297    2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 03:57:20.851664 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:20.851611    2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 03:57:20.851725 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:20.851684    2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-160.ec2.internal\" not found"
Oct 30 03:57:20.854106 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.854089    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:57:20.857101 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.857085    2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:57:20.932600 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.932528    2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-160.ec2.internal"]
Oct 30 03:57:20.932600 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.932598    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:57:20.933568 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.933552    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:57:20.933650 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.933580    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:57:20.933650 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.933590    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:57:20.935653 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.935638    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:57:20.935785 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.935763    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal"
Oct 30 03:57:20.935829 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.935802    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:57:20.937958 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.937939    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:57:20.938043 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.937971    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:57:20.938043 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.937970    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:57:20.938043 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.937985    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:57:20.938043 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.938001    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:57:20.938043 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.938020    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:57:20.940196 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.940181    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-160.ec2.internal"
Oct 30 03:57:20.940304 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.940208    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:57:20.941232 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.941213    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:57:20.941318 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.941261    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:57:20.941318 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.941281    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:57:20.951074 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.951062    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:57:20.951848 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.951831    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:57:20.951920 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.951857    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:57:20.951920 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.951867    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:57:20.951920 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.951884    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-160.ec2.internal"
Oct 30 03:57:20.967578 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.967557    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5bcce5d87346fa9cc82e9cdb4fa012bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal\" (UID: \"5bcce5d87346fa9cc82e9cdb4fa012bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal"
Oct 30 03:57:20.974962 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:20.974949    2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-160.ec2.internal"
Oct 30 03:57:20.975013 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:20.974968    2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-160.ec2.internal\": node \"ip-10-0-138-160.ec2.internal\" not found"
Oct 30 03:57:20.982541 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:20.982525    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-160.ec2.internal\" not found" node="ip-10-0-138-160.ec2.internal"
Oct 30 03:57:20.985126 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:20.985111    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-160.ec2.internal\" not found" node="ip-10-0-138-160.ec2.internal"
Oct 30 03:57:21.016083 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:21.016066    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-160.ec2.internal\" not found"
Oct 30 03:57:21.067675 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:21.067653    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5bcce5d87346fa9cc82e9cdb4fa012bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal\" (UID: \"5bcce5d87346fa9cc82e9cdb4fa012bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal"
Oct 30 03:57:21.067734 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:21.067686    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bcce5d87346fa9cc82e9cdb4fa012bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal\" (UID: \"5bcce5d87346fa9cc82e9cdb4fa012bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal"
Oct 30 03:57:21.067734 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:21.067696    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5bcce5d87346fa9cc82e9cdb4fa012bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal\" (UID: \"5bcce5d87346fa9cc82e9cdb4fa012bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal"
Oct 30 03:57:21.067734 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:21.067701    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc4b060adf5b640eabc59287108542f2-config\") pod \"kube-apiserver-proxy-ip-10-0-138-160.ec2.internal\" (UID: \"bc4b060adf5b640eabc59287108542f2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-160.ec2.internal"
Oct 30 03:57:21.116353 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:21.116331    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-160.ec2.internal\" not found"
Oct 30 03:57:21.168107 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:21.168087    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc4b060adf5b640eabc59287108542f2-config\") pod \"kube-apiserver-proxy-ip-10-0-138-160.ec2.internal\" (UID: \"bc4b060adf5b640eabc59287108542f2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-160.ec2.internal"
Oct 30 03:57:21.168193 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:21.168119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bcce5d87346fa9cc82e9cdb4fa012bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal\" (UID: \"5bcce5d87346fa9cc82e9cdb4fa012bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal"
Oct 30 03:57:21.168193 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:21.168151    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bcce5d87346fa9cc82e9cdb4fa012bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal\" (UID: \"5bcce5d87346fa9cc82e9cdb4fa012bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal"
Oct 30 03:57:21.168193 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:21.168169    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc4b060adf5b640eabc59287108542f2-config\") pod \"kube-apiserver-proxy-ip-10-0-138-160.ec2.internal\" (UID: \"bc4b060adf5b640eabc59287108542f2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-160.ec2.internal"
Oct 30 03:57:21.216859 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:21.216813    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-160.ec2.internal\" not found"
Oct 30 03:57:21.286814 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:21.286792    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal"
Oct 30 03:57:21.289993 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:21.289975    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-160.ec2.internal"
Oct 30 03:57:21.317532 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:21.317509    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-160.ec2.internal\" not found"
Oct 30 03:57:21.418135 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:21.418100    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-160.ec2.internal\" not found"
Oct 30 03:57:21.518635 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:21.518561    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-160.ec2.internal\" not found"
Oct 30 03:57:21.619147 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:21.619119    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-160.ec2.internal\" not found"
Oct 30 03:57:21.686709 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:21.686689    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 03:57:21.687066 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:21.686857    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 03:57:21.687066 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:21.686864    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 03:57:21.719995 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:21.719951    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-160.ec2.internal\" not found"
Oct 30 03:57:21.794439 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:21.794342    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 03:52:20 +0000 UTC" deadline="2028-08-06 11:41:49.580610784 +0000 UTC"
Oct 30 03:57:21.794439 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:21.794379    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15511h44m27.786235264s"
Oct 30 03:57:21.820295 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:21.820267    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-160.ec2.internal\" not found"
Oct 30 03:57:21.842721 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:21.842692    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bcce5d87346fa9cc82e9cdb4fa012bd.slice/crio-be470c5342fe3f1ea2b0543ae821eed994fb7a593b4b1faa4da163a0be40341b WatchSource:0}: Error finding container be470c5342fe3f1ea2b0543ae821eed994fb7a593b4b1faa4da163a0be40341b: Status 404 returned error can't find the container with id be470c5342fe3f1ea2b0543ae821eed994fb7a593b4b1faa4da163a0be40341b
Oct 30 03:57:21.844299 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:21.844263    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc4b060adf5b640eabc59287108542f2.slice/crio-dcdefb467a535c40a92e290ebf30908bdb38a82eed0d2db9ec8de3b3c454f2da WatchSource:0}: Error finding container dcdefb467a535c40a92e290ebf30908bdb38a82eed0d2db9ec8de3b3c454f2da: Status 404 returned error can't find the container with id dcdefb467a535c40a92e290ebf30908bdb38a82eed0d2db9ec8de3b3c454f2da
Oct 30 03:57:21.851605 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:21.851582    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:57:21.921295 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:21.921276    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-160.ec2.internal\" not found"
Oct 30 03:57:21.986673 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:21.986638    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:57:21.987839 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:21.987813    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:57:21.987943 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:21.987852    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:57:21.987943 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:21.987864    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:57:22.021791 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:22.021771    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-160.ec2.internal\" not found"
Oct 30 03:57:22.045126 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.045075    2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:57:22.067162 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.067133    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal"
Oct 30 03:57:22.090312 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.090294    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 03:57:22.090414 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.090399    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-160.ec2.internal"
Oct 30 03:57:22.109753 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.109735    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 03:57:22.540672 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.540592    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:57:22.708157 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.708128    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:57:22.734480 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.734454    2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:57:22.737366 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.737346    2578 apiserver.go:52] "Watching apiserver"
Oct 30 03:57:22.767304 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.767277    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 03:57:22.772701 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.772671    2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 03:57:22.773117 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.773087    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-lrcg5","openshift-multus/network-metrics-daemon-dpdr4","kube-system/kube-apiserver-proxy-ip-10-0-138-160.ec2.internal","openshift-dns/node-resolver-8g477","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal","openshift-network-operator/iptables-alerter-ttsr6","openshift-ovn-kubernetes/ovnkube-node-244hd","kube-system/konnectivity-agent-vnv9w","openshift-multus/multus-additional-cni-plugins-9nxvt","openshift-multus/multus-zl7qk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4","openshift-cluster-node-tuning-operator/tuned-b88ss","openshift-network-diagnostics/network-check-target-9mktl","kube-system/global-pull-secret-syncer-cbfm8"]
Oct 30 03:57:22.776266 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.776223    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.778085 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.778065    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ttsr6"
Oct 30 03:57:22.780335 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.780318    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.782895 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.782878    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8g477"
Oct 30 03:57:22.784855 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.784838    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:22.784936 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:22.784912    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mktl" podUID="d5e46634-9969-4d78-bb91-e598e950b1b0"
Oct 30 03:57:22.787385 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.787365    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:57:22.790186 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.790169    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:57:22.793144 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.792828    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:22.793144 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:22.792906    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpdr4" podUID="4dec34b4-fb69-459e-b130-ca1716d23765"
Oct 30 03:57:22.793386 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.793367    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 03:57:22.793481 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.793466    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 03:57:22.793737 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.793718    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 03:57:22.793737 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.793718    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 03:57:22.793898 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.793811    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 03:57:22.795807 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.795789    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 03:57:22.796094 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.796079    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.799158 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.798522    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 03:57:22.799158 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.798637    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lrcg5"
Oct 30 03:57:22.800286 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.800228    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vnv9w"
Oct 30 03:57:22.803063 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.802906    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dgvb4"]
Oct 30 03:57:22.803063 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.802972    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:22.803063 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:22.803036    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbfm8" podUID="bad69b26-9de8-4c2b-b19c-8a96271f5c86"
Oct 30 03:57:22.805756 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.805739    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.819655 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.819410    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 03:57:22.819655 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.819485    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rs54w\""
Oct 30 03:57:22.819759 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.819705    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 03:57:22.819803 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.819762    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h7p9v\""
Oct 30 03:57:22.819847 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.819801    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vm8lh\""
Oct 30 03:57:22.819911 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.819894    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4xznk\""
Oct 30 03:57:22.819990 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.819967    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kwf82\""
Oct 30 03:57:22.820500 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.820477    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-skmdf\""
Oct 30 03:57:22.820631 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.820617    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 03:57:22.820766 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.820733    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w4blj\""
Oct 30 03:57:22.820872 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.820780    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-dnvhh\""
Oct 30 03:57:22.820872 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.820780    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-smvt7\""
Oct 30 03:57:22.826335 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.826314    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 03:57:22.826431 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.826373    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m7m9p"
Oct 30 03:57:22.837232 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.837147    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-160.ec2.internal" event={"ID":"bc4b060adf5b640eabc59287108542f2","Type":"ContainerStarted","Data":"dcdefb467a535c40a92e290ebf30908bdb38a82eed0d2db9ec8de3b3c454f2da"}
Oct 30 03:57:22.839415 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.839381    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal" event={"ID":"5bcce5d87346fa9cc82e9cdb4fa012bd","Type":"ContainerStarted","Data":"be470c5342fe3f1ea2b0543ae821eed994fb7a593b4b1faa4da163a0be40341b"}
Oct 30 03:57:22.847920 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.847898    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m7m9p"
Oct 30 03:57:22.855615 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.855552    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 03:57:22.855872 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.855806    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 03:57:22.855957 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.855873    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 03:57:22.856696 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.856678    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 03:57:22.856787 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.856714    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 03:57:22.856787 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.856732    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 03:57:22.856787 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.856782    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 03:57:22.856787 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.856785    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 03:57:22.856993 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.856809    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:57:22.856993 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.856824    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 03:57:22.856993 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.856859    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 03:57:22.856993 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.856888    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 03:57:22.856993 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.856676    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 03:57:22.856993 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.856925    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 03:57:22.856993 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.856951    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 03:57:22.856993 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.856958    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 03:57:22.856993 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.856984    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:57:22.856993 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.856739    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 03:57:22.857405 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.856926    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:57:22.857405 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.857022    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:57:22.857405 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.857042    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 03:57:22.868700 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.868683    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 03:57:22.878969 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.878373    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/455076cd-b484-44b6-8a1b-b4c3c0dccab8-serviceca\") pod \"node-ca-lrcg5\" (UID: \"455076cd-b484-44b6-8a1b-b4c3c0dccab8\") " pod="openshift-image-registry/node-ca-lrcg5"
Oct 30 03:57:22.878969 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.878794    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-tls\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.878969 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.878824    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-sysctl-d\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.878969 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.878860    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:57:22.878969 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.878895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-var-lib-cni-multus\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.878969 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.878927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bad69b26-9de8-4c2b-b19c-8a96271f5c86-kubelet-config\") pod \"global-pull-secret-syncer-cbfm8\" (UID: \"bad69b26-9de8-4c2b-b19c-8a96271f5c86\") " pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:22.878969 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.878969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30e672ce-27b7-4997-a887-b834e6a6797d-sys\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.879336 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.879001    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-run\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.879336 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.879040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-var-lib-openvswitch\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.879336 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.879073    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-node-log\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.879336 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.879104    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-accelerators-collector-config\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.879336 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.879131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-systemd\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.879336 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.879156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-run-k8s-cni-cncf-io\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.879336 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.879190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-multus-conf-dir\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.879336 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.879218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grbsb\" (UniqueName: \"kubernetes.io/projected/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-kube-api-access-grbsb\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.879336 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.879259    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/97de5e90-d514-4b8c-8917-2e9894978aaa-ovnkube-script-lib\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.879336 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.879293    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z24jg\" (UniqueName: \"kubernetes.io/projected/97de5e90-d514-4b8c-8917-2e9894978aaa-kube-api-access-z24jg\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.879336 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.879317    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-sys-fs\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:57:22.879779 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.879344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-sys\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.879779 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.879372    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-host\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.879779 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.879409    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-cni-binary-copy\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.879779 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.879437    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bad69b26-9de8-4c2b-b19c-8a96271f5c86-dbus\") pod \"global-pull-secret-syncer-cbfm8\" (UID: \"bad69b26-9de8-4c2b-b19c-8a96271f5c86\") " pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:22.879779 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.879465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-device-dir\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:57:22.879779 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.879492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/64f1a482-331f-4490-b03f-f41facd80cfe-etc-tuned\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.879779 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.879522    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-kubelet\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.879779 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.879549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/455076cd-b484-44b6-8a1b-b4c3c0dccab8-host\") pod \"node-ca-lrcg5\" (UID: \"455076cd-b484-44b6-8a1b-b4c3c0dccab8\") " pod="openshift-image-registry/node-ca-lrcg5"
Oct 30 03:57:22.879779 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.879576    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-run-netns\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.879779 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.879599    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-etc-openvswitch\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.879779 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.879632    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dec34b4-fb69-459e-b130-ca1716d23765-metrics-certs\") pod \"network-metrics-daemon-dpdr4\" (UID: \"4dec34b4-fb69-459e-b130-ca1716d23765\") " pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:22.882271 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.880291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bad69b26-9de8-4c2b-b19c-8a96271f5c86-original-pull-secret\") pod \"global-pull-secret-syncer-cbfm8\" (UID: \"bad69b26-9de8-4c2b-b19c-8a96271f5c86\") " pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:22.882271 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.880376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxh5g\" (UniqueName: \"kubernetes.io/projected/baf40c59-c04b-4c27-b080-6c55887c5c4c-kube-api-access-bxh5g\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:57:22.882271 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.880405    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-system-cni-dir\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:57:22.882271 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.880433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-multus-daemon-config\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.882271 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.880474    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-etc-kubernetes\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.882271 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.880530    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/97de5e90-d514-4b8c-8917-2e9894978aaa-env-overrides\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.882271 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.880552    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg642\" (UniqueName: \"kubernetes.io/projected/455076cd-b484-44b6-8a1b-b4c3c0dccab8-kube-api-access-zg642\") pod \"node-ca-lrcg5\" (UID: \"455076cd-b484-44b6-8a1b-b4c3c0dccab8\") " pod="openshift-image-registry/node-ca-lrcg5"
Oct 30 03:57:22.882271 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.880579    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-wtmp\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.882271 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.880612    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30e672ce-27b7-4997-a887-b834e6a6797d-metrics-client-ca\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.882271 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.880650    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.882271 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.880718    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-cnibin\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:57:22.882271 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.880776    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:57:22.882271 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.880829    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-system-cni-dir\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.882271 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.880878    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.882271 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.880930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:57:22.882271 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.880952    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30e672ce-27b7-4997-a887-b834e6a6797d-root\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.882848 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.880982    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hl75\" (UniqueName: \"kubernetes.io/projected/30e672ce-27b7-4997-a887-b834e6a6797d-kube-api-access-5hl75\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.882848 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.881011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-kubernetes\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.882848 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.881040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ade9bd13-6259-48aa-92b2-52f875b960e8-hosts-file\") pod \"node-resolver-8g477\" (UID: \"ade9bd13-6259-48aa-92b2-52f875b960e8\") " pod="openshift-dns/node-resolver-8g477"
Oct 30 03:57:22.882848 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.881116    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-run-netns\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.882848 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.881157    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-run-ovn\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.882848 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.881206    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-log-socket\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.882848 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.881234    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67ea77e7-74cc-4bc6-ae16-5a5a797f5097-host-slash\") pod \"iptables-alerter-ttsr6\" (UID: \"67ea77e7-74cc-4bc6-ae16-5a5a797f5097\") " pod="openshift-network-operator/iptables-alerter-ttsr6"
Oct 30 03:57:22.882848 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.881277    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfdvg\" (UniqueName: \"kubernetes.io/projected/67ea77e7-74cc-4bc6-ae16-5a5a797f5097-kube-api-access-zfdvg\") pod \"iptables-alerter-ttsr6\" (UID: \"67ea77e7-74cc-4bc6-ae16-5a5a797f5097\") " pod="openshift-network-operator/iptables-alerter-ttsr6"
Oct 30 03:57:22.882848 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.881305    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-etc-selinux\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:57:22.882848 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.881331    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-sysctl-conf\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.882848 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.881389    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwtgk\" (UniqueName: \"kubernetes.io/projected/d5e46634-9969-4d78-bb91-e598e950b1b0-kube-api-access-jwtgk\") pod \"network-check-target-9mktl\" (UID: \"d5e46634-9969-4d78-bb91-e598e950b1b0\") " pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:22.882848 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.881417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kq65\" (UniqueName: \"kubernetes.io/projected/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-kube-api-access-6kq65\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:57:22.882848 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.881492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-cnibin\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.882848 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.881550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-slash\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.882848 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.881577    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-run-openvswitch\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.882848 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.881620    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/97de5e90-d514-4b8c-8917-2e9894978aaa-ovn-node-metrics-cert\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.883523 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.881657    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-lib-modules\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.883523 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.881679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade9bd13-6259-48aa-92b2-52f875b960e8-tmp-dir\") pod \"node-resolver-8g477\" (UID: \"ade9bd13-6259-48aa-92b2-52f875b960e8\") " pod="openshift-dns/node-resolver-8g477"
Oct 30 03:57:22.883523 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.881729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:57:22.883523 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.881755    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-hostroot\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.883523 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.882024    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/97de5e90-d514-4b8c-8917-2e9894978aaa-ovnkube-config\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.883523 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.882082    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qr8r\" (UniqueName: \"kubernetes.io/projected/4dec34b4-fb69-459e-b130-ca1716d23765-kube-api-access-4qr8r\") pod \"network-metrics-daemon-dpdr4\" (UID: \"4dec34b4-fb69-459e-b130-ca1716d23765\") " pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:22.883523 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.882131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64f1a482-331f-4490-b03f-f41facd80cfe-tmp\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.883523 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.882195    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rszkc\" (UniqueName: \"kubernetes.io/projected/64f1a482-331f-4490-b03f-f41facd80cfe-kube-api-access-rszkc\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.883523 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.882219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/67ea77e7-74cc-4bc6-ae16-5a5a797f5097-iptables-alerter-script\") pod \"iptables-alerter-ttsr6\" (UID: \"67ea77e7-74cc-4bc6-ae16-5a5a797f5097\") " pod="openshift-network-operator/iptables-alerter-ttsr6"
Oct 30 03:57:22.883523 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.882310    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-registration-dir\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:57:22.883523 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.882356    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-textfile\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.883523 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.882385    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e385d23f-6371-4f69-97b1-6a10303c048f-konnectivity-ca\") pod \"konnectivity-agent-vnv9w\" (UID: \"e385d23f-6371-4f69-97b1-6a10303c048f\") " pod="kube-system/konnectivity-agent-vnv9w"
Oct 30 03:57:22.883523 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.882452    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-var-lib-cni-bin\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.883523 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.882482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-var-lib-kubelet\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.883523 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.882498    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-cni-bin\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.883523 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.882526    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-sysconfig\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.884167 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.882559    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-var-lib-kubelet\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.884167 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.882595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-run-multus-certs\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.884167 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.882622    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvzdn\" (UniqueName: \"kubernetes.io/projected/ade9bd13-6259-48aa-92b2-52f875b960e8-kube-api-access-lvzdn\") pod \"node-resolver-8g477\" (UID: \"ade9bd13-6259-48aa-92b2-52f875b960e8\") " pod="openshift-dns/node-resolver-8g477"
Oct 30 03:57:22.884167 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.882654    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-systemd-units\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.884167 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.882684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-run-systemd\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.884167 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.882705    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-socket-dir\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:57:22.884167 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.882722    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-modprobe-d\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.884167 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.882744    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:57:22.884167 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.882869    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-cni-netd\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.884167 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.882899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e385d23f-6371-4f69-97b1-6a10303c048f-agent-certs\") pod \"konnectivity-agent-vnv9w\" (UID: \"e385d23f-6371-4f69-97b1-6a10303c048f\") " pod="kube-system/konnectivity-agent-vnv9w"
Oct 30 03:57:22.884167 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.882918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-multus-cni-dir\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.884167 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.882943    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-multus-socket-dir-parent\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.884167 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.882969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.886900 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.886883    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 03:57:22.887041 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.887010    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 03:57:22.887295 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.887280    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8s8mj\""
Oct 30 03:57:22.984111 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-var-lib-cni-bin\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.984278 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-var-lib-kubelet\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.984278 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-cni-bin\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.984278 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984181    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-sysconfig\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.984278 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-var-lib-kubelet\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.984278 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984218    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-var-lib-cni-bin\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.984278 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984233    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-run-multus-certs\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.984278 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvzdn\" (UniqueName: \"kubernetes.io/projected/ade9bd13-6259-48aa-92b2-52f875b960e8-kube-api-access-lvzdn\") pod \"node-resolver-8g477\" (UID: \"ade9bd13-6259-48aa-92b2-52f875b960e8\") " pod="openshift-dns/node-resolver-8g477"
Oct 30 03:57:22.984563 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984286    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-cni-bin\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.984563 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984326    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-systemd-units\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.984563 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984330    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-var-lib-kubelet\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.984563 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-run-systemd\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.984563 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984378    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-run-multus-certs\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.984563 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-socket-dir\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:57:22.984563 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984410    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-modprobe-d\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.984563 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984433    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-sysconfig\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.984563 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984435    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:57:22.984563 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-cni-netd\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.984563 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e385d23f-6371-4f69-97b1-6a10303c048f-agent-certs\") pod \"konnectivity-agent-vnv9w\" (UID: \"e385d23f-6371-4f69-97b1-6a10303c048f\") " pod="kube-system/konnectivity-agent-vnv9w"
Oct 30 03:57:22.984563 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984502    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-multus-cni-dir\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.984563 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984519    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-multus-socket-dir-parent\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.984563 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984557    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.985118 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/455076cd-b484-44b6-8a1b-b4c3c0dccab8-serviceca\") pod \"node-ca-lrcg5\" (UID: \"455076cd-b484-44b6-8a1b-b4c3c0dccab8\") " pod="openshift-image-registry/node-ca-lrcg5"
Oct 30 03:57:22.985118 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-tls\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.985118 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-sysctl-d\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.985118 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:57:22.985118 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984654    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-var-lib-cni-multus\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.985118 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984669    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bad69b26-9de8-4c2b-b19c-8a96271f5c86-kubelet-config\") pod \"global-pull-secret-syncer-cbfm8\" (UID: \"bad69b26-9de8-4c2b-b19c-8a96271f5c86\") " pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:22.985118 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30e672ce-27b7-4997-a887-b834e6a6797d-sys\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.985118 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984716    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-run\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.985118 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-var-lib-openvswitch\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.985118 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984759    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-systemd-units\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.985118 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984763    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-node-log\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.985118 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984794    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-accelerators-collector-config\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.985118 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984820    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-run-systemd\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.985118 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984834    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-systemd\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.985118 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984851    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-run-k8s-cni-cncf-io\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.985118 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984867    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-multus-conf-dir\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.985118 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grbsb\" (UniqueName: \"kubernetes.io/projected/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-kube-api-access-grbsb\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.985723 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/97de5e90-d514-4b8c-8917-2e9894978aaa-ovnkube-script-lib\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.985723 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984924    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-socket-dir\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:57:22.985723 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z24jg\" (UniqueName: \"kubernetes.io/projected/97de5e90-d514-4b8c-8917-2e9894978aaa-kube-api-access-z24jg\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.985723 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-sys-fs\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:57:22.985723 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.984998    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-sys\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.985723 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985037    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-host\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.985723 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-cni-binary-copy\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.985723 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985093    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bad69b26-9de8-4c2b-b19c-8a96271f5c86-dbus\") pod \"global-pull-secret-syncer-cbfm8\" (UID: \"bad69b26-9de8-4c2b-b19c-8a96271f5c86\") " pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:22.985723 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-device-dir\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:57:22.985723 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985185    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-device-dir\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:57:22.985723 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985181    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-var-lib-kubelet\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.985723 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985318    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-modprobe-d\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.985723 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985336    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:57:22.985723 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985349    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/64f1a482-331f-4490-b03f-f41facd80cfe-etc-tuned\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.985723 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985366    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-cni-netd\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.985723 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-kubelet\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.985723 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/455076cd-b484-44b6-8a1b-b4c3c0dccab8-host\") pod \"node-ca-lrcg5\" (UID: \"455076cd-b484-44b6-8a1b-b4c3c0dccab8\") " pod="openshift-image-registry/node-ca-lrcg5"
Oct 30 03:57:22.985723 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-run-netns\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.986786 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985465    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-etc-openvswitch\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.986786 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dec34b4-fb69-459e-b130-ca1716d23765-metrics-certs\") pod \"network-metrics-daemon-dpdr4\" (UID: \"4dec34b4-fb69-459e-b130-ca1716d23765\") " pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:22.986786 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985517    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bad69b26-9de8-4c2b-b19c-8a96271f5c86-original-pull-secret\") pod \"global-pull-secret-syncer-cbfm8\" (UID: \"bad69b26-9de8-4c2b-b19c-8a96271f5c86\") " pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:22.986786 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxh5g\" (UniqueName: \"kubernetes.io/projected/baf40c59-c04b-4c27-b080-6c55887c5c4c-kube-api-access-bxh5g\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:57:22.986786 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-system-cni-dir\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:57:22.986786 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985605    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-multus-daemon-config\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.986786 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-etc-kubernetes\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.986786 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985660    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30e672ce-27b7-4997-a887-b834e6a6797d-sys\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.986786 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985672    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/97de5e90-d514-4b8c-8917-2e9894978aaa-env-overrides\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.986786 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985671    2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 03:57:22.986786 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985873    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-multus-cni-dir\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.986786 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985882    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-multus-socket-dir-parent\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.986786 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985940    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-systemd\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.986786 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985956    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.986786 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.985990    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-run\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.986786 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986032    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-var-lib-openvswitch\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.986786 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986078    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-node-log\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.986786 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986140    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/455076cd-b484-44b6-8a1b-b4c3c0dccab8-host\") pod \"node-ca-lrcg5\" (UID: \"455076cd-b484-44b6-8a1b-b4c3c0dccab8\") " pod="openshift-image-registry/node-ca-lrcg5"
Oct 30 03:57:22.987628 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986201    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bad69b26-9de8-4c2b-b19c-8a96271f5c86-dbus\") pod \"global-pull-secret-syncer-cbfm8\" (UID: \"bad69b26-9de8-4c2b-b19c-8a96271f5c86\") " pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:22.987628 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986219    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-kubelet\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.987628 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986234    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-var-lib-cni-multus\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.987628 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986281    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-multus-conf-dir\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.987628 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986286    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg642\" (UniqueName: \"kubernetes.io/projected/455076cd-b484-44b6-8a1b-b4c3c0dccab8-kube-api-access-zg642\") pod \"node-ca-lrcg5\" (UID: \"455076cd-b484-44b6-8a1b-b4c3c0dccab8\") " pod="openshift-image-registry/node-ca-lrcg5"
Oct 30 03:57:22.987628 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986308    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bad69b26-9de8-4c2b-b19c-8a96271f5c86-kubelet-config\") pod \"global-pull-secret-syncer-cbfm8\" (UID: \"bad69b26-9de8-4c2b-b19c-8a96271f5c86\") " pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:22.987628 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986320    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-run-k8s-cni-cncf-io\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.987628 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-wtmp\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.987628 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986413    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-sysctl-d\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.987628 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986439    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-wtmp\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.987628 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30e672ce-27b7-4997-a887-b834e6a6797d-metrics-client-ca\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.987628 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986473    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.987628 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986488    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-system-cni-dir\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:57:22.987628 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986522    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-etc-openvswitch\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.987628 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-cnibin\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:57:22.987628 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986562    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-run-netns\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.987628 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986565    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:57:22.988405 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-system-cni-dir\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.988405 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986631    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.988405 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986659    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:57:22.988405 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30e672ce-27b7-4997-a887-b834e6a6797d-root\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.988405 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986699    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-etc-kubernetes\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.988405 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hl75\" (UniqueName: \"kubernetes.io/projected/30e672ce-27b7-4997-a887-b834e6a6797d-kube-api-access-5hl75\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.988405 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986733    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-kubernetes\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.988405 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ade9bd13-6259-48aa-92b2-52f875b960e8-hosts-file\") pod \"node-resolver-8g477\" (UID: \"ade9bd13-6259-48aa-92b2-52f875b960e8\") " pod="openshift-dns/node-resolver-8g477"
Oct 30 03:57:22.988405 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:22.986772    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:57:22.988405 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986781    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-run-netns\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.988405 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-run-ovn\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.988405 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986828    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-log-socket\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.988405 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:22.986845    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dec34b4-fb69-459e-b130-ca1716d23765-metrics-certs podName:4dec34b4-fb69-459e-b130-ca1716d23765 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:23.486826027 +0000 UTC m=+3.116345770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4dec34b4-fb69-459e-b130-ca1716d23765-metrics-certs") pod "network-metrics-daemon-dpdr4" (UID: "4dec34b4-fb69-459e-b130-ca1716d23765") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:57:22.988405 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:22.986859    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 03:57:22.988405 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986864    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-log-socket\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.988405 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986876    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67ea77e7-74cc-4bc6-ae16-5a5a797f5097-host-slash\") pod \"iptables-alerter-ttsr6\" (UID: \"67ea77e7-74cc-4bc6-ae16-5a5a797f5097\") " pod="openshift-network-operator/iptables-alerter-ttsr6"
Oct 30 03:57:22.988405 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986886    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:57:22.988405 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfdvg\" (UniqueName: \"kubernetes.io/projected/67ea77e7-74cc-4bc6-ae16-5a5a797f5097-kube-api-access-zfdvg\") pod \"iptables-alerter-ttsr6\" (UID: \"67ea77e7-74cc-4bc6-ae16-5a5a797f5097\") " pod="openshift-network-operator/iptables-alerter-ttsr6"
Oct 30 03:57:22.989197 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.986908    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30e672ce-27b7-4997-a887-b834e6a6797d-root\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.989197 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:22.986959    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bad69b26-9de8-4c2b-b19c-8a96271f5c86-original-pull-secret podName:bad69b26-9de8-4c2b-b19c-8a96271f5c86 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:23.486903468 +0000 UTC m=+3.116423206 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bad69b26-9de8-4c2b-b19c-8a96271f5c86-original-pull-secret") pod "global-pull-secret-syncer-cbfm8" (UID: "bad69b26-9de8-4c2b-b19c-8a96271f5c86") : object "kube-system"/"original-pull-secret" not registered
Oct 30 03:57:22.989197 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-etc-selinux\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:57:22.989197 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-sysctl-conf\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.989197 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwtgk\" (UniqueName: \"kubernetes.io/projected/d5e46634-9969-4d78-bb91-e598e950b1b0-kube-api-access-jwtgk\") pod \"network-check-target-9mktl\" (UID: \"d5e46634-9969-4d78-bb91-e598e950b1b0\") " pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:22.989197 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987072    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-kubernetes\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.989197 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987124    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ade9bd13-6259-48aa-92b2-52f875b960e8-hosts-file\") pod \"node-resolver-8g477\" (UID: \"ade9bd13-6259-48aa-92b2-52f875b960e8\") " pod="openshift-dns/node-resolver-8g477"
Oct 30 03:57:22.989197 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987135    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kq65\" (UniqueName: \"kubernetes.io/projected/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-kube-api-access-6kq65\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:57:22.989197 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987143    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67ea77e7-74cc-4bc6-ae16-5a5a797f5097-host-slash\") pod \"iptables-alerter-ttsr6\" (UID: \"67ea77e7-74cc-4bc6-ae16-5a5a797f5097\") " pod="openshift-network-operator/iptables-alerter-ttsr6"
Oct 30 03:57:22.989197 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987161    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-run-netns\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.989197 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-cnibin\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.989197 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-slash\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.989197 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987211    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-cnibin\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:57:22.989197 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-run-openvswitch\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.989197 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987327    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/97de5e90-d514-4b8c-8917-2e9894978aaa-ovn-node-metrics-cert\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.989197 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987352    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-lib-modules\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.989197 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade9bd13-6259-48aa-92b2-52f875b960e8-tmp-dir\") pod \"node-resolver-8g477\" (UID: \"ade9bd13-6259-48aa-92b2-52f875b960e8\") " pod="openshift-dns/node-resolver-8g477"
Oct 30 03:57:22.989951 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:57:22.989951 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987463    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-hostroot\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.989951 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987478    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-etc-selinux\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:57:22.989951 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/97de5e90-d514-4b8c-8917-2e9894978aaa-ovnkube-config\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.989951 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987516    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qr8r\" (UniqueName: \"kubernetes.io/projected/4dec34b4-fb69-459e-b130-ca1716d23765-kube-api-access-4qr8r\") pod \"network-metrics-daemon-dpdr4\" (UID: \"4dec34b4-fb69-459e-b130-ca1716d23765\") " pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:22.989951 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987572    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-run-ovn\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.989951 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987587    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-sysctl-conf\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.989951 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64f1a482-331f-4490-b03f-f41facd80cfe-tmp\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.989951 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rszkc\" (UniqueName: \"kubernetes.io/projected/64f1a482-331f-4490-b03f-f41facd80cfe-kube-api-access-rszkc\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.989951 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987644    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-system-cni-dir\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.989951 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987658    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/67ea77e7-74cc-4bc6-ae16-5a5a797f5097-iptables-alerter-script\") pod \"iptables-alerter-ttsr6\" (UID: \"67ea77e7-74cc-4bc6-ae16-5a5a797f5097\") " pod="openshift-network-operator/iptables-alerter-ttsr6"
Oct 30 03:57:22.989951 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987686    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-registration-dir\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:57:22.989951 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987693    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.989951 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987712    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-textfile\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.989951 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.987737    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e385d23f-6371-4f69-97b1-6a10303c048f-konnectivity-ca\") pod \"konnectivity-agent-vnv9w\" (UID: \"e385d23f-6371-4f69-97b1-6a10303c048f\") " pod="kube-system/konnectivity-agent-vnv9w"
Oct 30 03:57:22.989951 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.988200    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e385d23f-6371-4f69-97b1-6a10303c048f-konnectivity-ca\") pod \"konnectivity-agent-vnv9w\" (UID: \"e385d23f-6371-4f69-97b1-6a10303c048f\") " pod="kube-system/konnectivity-agent-vnv9w"
Oct 30 03:57:22.989951 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.988235    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/97de5e90-d514-4b8c-8917-2e9894978aaa-ovnkube-script-lib\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.990603 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.988297    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-registration-dir\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:57:22.990603 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.988342    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-sys\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.990603 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.988372    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-sys-fs\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:57:22.990603 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.988449    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-lib-modules\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.990603 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.988622    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-cnibin\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.990603 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.988654    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-slash\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.990603 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.988686    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-run-openvswitch\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.990603 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.988861    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/67ea77e7-74cc-4bc6-ae16-5a5a797f5097-iptables-alerter-script\") pod \"iptables-alerter-ttsr6\" (UID: \"67ea77e7-74cc-4bc6-ae16-5a5a797f5097\") " pod="openshift-network-operator/iptables-alerter-ttsr6"
Oct 30 03:57:22.990603 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.989082    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/97de5e90-d514-4b8c-8917-2e9894978aaa-env-overrides\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.990603 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.989599    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-accelerators-collector-config\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.990603 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.989715    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:57:22.990603 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.989833    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-host\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.990603 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.989888    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-hostroot\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.992130 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.992106    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade9bd13-6259-48aa-92b2-52f875b960e8-tmp-dir\") pod \"node-resolver-8g477\" (UID: \"ade9bd13-6259-48aa-92b2-52f875b960e8\") " pod="openshift-dns/node-resolver-8g477"
Oct 30 03:57:22.992232 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.992170    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-multus-daemon-config\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.992367 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.992348    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-cni-binary-copy\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:22.992434 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.992397    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:57:22.992631 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.992568    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:57:22.992631 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.992583    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30e672ce-27b7-4997-a887-b834e6a6797d-metrics-client-ca\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.992743 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.992687    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/455076cd-b484-44b6-8a1b-b4c3c0dccab8-serviceca\") pod \"node-ca-lrcg5\" (UID: \"455076cd-b484-44b6-8a1b-b4c3c0dccab8\") " pod="openshift-image-registry/node-ca-lrcg5"
Oct 30 03:57:22.993460 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.993314    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.993460 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.993391    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e385d23f-6371-4f69-97b1-6a10303c048f-agent-certs\") pod \"konnectivity-agent-vnv9w\" (UID: \"e385d23f-6371-4f69-97b1-6a10303c048f\") " pod="kube-system/konnectivity-agent-vnv9w"
Oct 30 03:57:22.993776 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.993664    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/64f1a482-331f-4490-b03f-f41facd80cfe-etc-tuned\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.993871 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.993852    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/97de5e90-d514-4b8c-8917-2e9894978aaa-ovnkube-config\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.994068 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.993983    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64f1a482-331f-4490-b03f-f41facd80cfe-tmp\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:22.994068 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.994033    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-textfile\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:22.994187 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.994148    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/97de5e90-d514-4b8c-8917-2e9894978aaa-ovn-node-metrics-cert\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:22.997169 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:22.997122    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-tls\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:23.007097 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.007075    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvzdn\" (UniqueName: \"kubernetes.io/projected/ade9bd13-6259-48aa-92b2-52f875b960e8-kube-api-access-lvzdn\") pod \"node-resolver-8g477\" (UID: \"ade9bd13-6259-48aa-92b2-52f875b960e8\") " pod="openshift-dns/node-resolver-8g477"
Oct 30 03:57:23.007414 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:23.007396    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 03:57:23.007492 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:23.007418    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 03:57:23.007492 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:23.007430    2578 projected.go:196] Error preparing data for projected volume kube-api-access-jwtgk for pod openshift-network-diagnostics/network-check-target-9mktl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:57:23.007575 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:23.007493    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e46634-9969-4d78-bb91-e598e950b1b0-kube-api-access-jwtgk podName:d5e46634-9969-4d78-bb91-e598e950b1b0 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:23.507474167 +0000 UTC m=+3.136993903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jwtgk" (UniqueName: "kubernetes.io/projected/d5e46634-9969-4d78-bb91-e598e950b1b0-kube-api-access-jwtgk") pod "network-check-target-9mktl" (UID: "d5e46634-9969-4d78-bb91-e598e950b1b0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:57:23.009112 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.009089    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z24jg\" (UniqueName: \"kubernetes.io/projected/97de5e90-d514-4b8c-8917-2e9894978aaa-kube-api-access-z24jg\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:23.010103 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.010083    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg642\" (UniqueName: \"kubernetes.io/projected/455076cd-b484-44b6-8a1b-b4c3c0dccab8-kube-api-access-zg642\") pod \"node-ca-lrcg5\" (UID: \"455076cd-b484-44b6-8a1b-b4c3c0dccab8\") " pod="openshift-image-registry/node-ca-lrcg5"
Oct 30 03:57:23.011676 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.011652    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rszkc\" (UniqueName: \"kubernetes.io/projected/64f1a482-331f-4490-b03f-f41facd80cfe-kube-api-access-rszkc\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:23.012339 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.012299    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grbsb\" (UniqueName: \"kubernetes.io/projected/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-kube-api-access-grbsb\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:23.012617 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.012596    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hl75\" (UniqueName: \"kubernetes.io/projected/30e672ce-27b7-4997-a887-b834e6a6797d-kube-api-access-5hl75\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:23.012822 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.012804    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qr8r\" (UniqueName: \"kubernetes.io/projected/4dec34b4-fb69-459e-b130-ca1716d23765-kube-api-access-4qr8r\") pod \"network-metrics-daemon-dpdr4\" (UID: \"4dec34b4-fb69-459e-b130-ca1716d23765\") " pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:23.015309 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.015266    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kq65\" (UniqueName: \"kubernetes.io/projected/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-kube-api-access-6kq65\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:57:23.015983 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.015961    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxh5g\" (UniqueName: \"kubernetes.io/projected/baf40c59-c04b-4c27-b080-6c55887c5c4c-kube-api-access-bxh5g\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:57:23.016176 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.016155    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfdvg\" (UniqueName: \"kubernetes.io/projected/67ea77e7-74cc-4bc6-ae16-5a5a797f5097-kube-api-access-zfdvg\") pod \"iptables-alerter-ttsr6\" (UID: \"67ea77e7-74cc-4bc6-ae16-5a5a797f5097\") " pod="openshift-network-operator/iptables-alerter-ttsr6"
Oct 30 03:57:23.107324 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.107250    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ttsr6"
Oct 30 03:57:23.109416 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.109391    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:23.114201 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.114185    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:57:23.116694 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.116556    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8g477"
Oct 30 03:57:23.129161 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.129141    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:57:23.137070 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:23.137046    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f1a482_331f_4490_b03f_f41facd80cfe.slice/crio-65981a3a7a1f54c0ec3366558cc8b4a359ed404252cac9947b82dbcb41164bc8 WatchSource:0}: Error finding container 65981a3a7a1f54c0ec3366558cc8b4a359ed404252cac9947b82dbcb41164bc8: Status 404 returned error can't find the container with id 65981a3a7a1f54c0ec3366558cc8b4a359ed404252cac9947b82dbcb41164bc8
Oct 30 03:57:23.137839 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:23.137791    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade9bd13_6259_48aa_92b2_52f875b960e8.slice/crio-0121ea1bb37a9e8348390e8ef77e92d2f83e3f7208c8afd915debcfe1131184d WatchSource:0}: Error finding container 0121ea1bb37a9e8348390e8ef77e92d2f83e3f7208c8afd915debcfe1131184d: Status 404 returned error can't find the container with id 0121ea1bb37a9e8348390e8ef77e92d2f83e3f7208c8afd915debcfe1131184d
Oct 30 03:57:23.138905 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:23.138816    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ea77e7_74cc_4bc6_ae16_5a5a797f5097.slice/crio-1b399c10fd0f25666919e4b322b41999ca07b978930fc36c37ce9e191d6bef95 WatchSource:0}: Error finding container 1b399c10fd0f25666919e4b322b41999ca07b978930fc36c37ce9e191d6bef95: Status 404 returned error can't find the container with id 1b399c10fd0f25666919e4b322b41999ca07b978930fc36c37ce9e191d6bef95
Oct 30 03:57:23.140602 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:23.140578    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97de5e90_d514_4b8c_8917_2e9894978aaa.slice/crio-e99ebdc945a00b3970de14be54f3eabc56a7bfaed9413e043cc5311a161e1fb2 WatchSource:0}: Error finding container e99ebdc945a00b3970de14be54f3eabc56a7bfaed9413e043cc5311a161e1fb2: Status 404 returned error can't find the container with id e99ebdc945a00b3970de14be54f3eabc56a7bfaed9413e043cc5311a161e1fb2
Oct 30 03:57:23.145129 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.145113    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zl7qk"
Oct 30 03:57:23.146626 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.146607    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:57:23.148753 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.148738    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lrcg5"
Oct 30 03:57:23.153507 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:23.153489    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf40c59_c04b_4c27_b080_6c55887c5c4c.slice/crio-78e7047f49c55034afe2ea6bda525cea7236ce655fbab85b43eb933b778551b1 WatchSource:0}: Error finding container 78e7047f49c55034afe2ea6bda525cea7236ce655fbab85b43eb933b778551b1: Status 404 returned error can't find the container with id 78e7047f49c55034afe2ea6bda525cea7236ce655fbab85b43eb933b778551b1
Oct 30 03:57:23.154716 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.154692    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vnv9w"
Oct 30 03:57:23.158771 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:23.158749    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9f3a00_e77c_4946_b5bb_3a0cb171b83e.slice/crio-5523a203c9a2d75d7b7edb12c5dd54ef4c5676cc461ba54e3b38a768577c6ef0 WatchSource:0}: Error finding container 5523a203c9a2d75d7b7edb12c5dd54ef4c5676cc461ba54e3b38a768577c6ef0: Status 404 returned error can't find the container with id 5523a203c9a2d75d7b7edb12c5dd54ef4c5676cc461ba54e3b38a768577c6ef0
Oct 30 03:57:23.163416 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:23.163396    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455076cd_b484_44b6_8a1b_b4c3c0dccab8.slice/crio-77952d06a35c50393f61276f061dec66cdaa317609434651004514d570532709 WatchSource:0}: Error finding container 77952d06a35c50393f61276f061dec66cdaa317609434651004514d570532709: Status 404 returned error can't find the container with id 77952d06a35c50393f61276f061dec66cdaa317609434651004514d570532709
Oct 30 03:57:23.164063 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:23.164012    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef4cdba_d13e_4e6b_a4e6_d722d58b55e8.slice/crio-c0ddfb16511a887da3f3bd6c6bf63ae5d336361e54089180428a74ab9ca9f6f9 WatchSource:0}: Error finding container c0ddfb16511a887da3f3bd6c6bf63ae5d336361e54089180428a74ab9ca9f6f9: Status 404 returned error can't find the container with id c0ddfb16511a887da3f3bd6c6bf63ae5d336361e54089180428a74ab9ca9f6f9
Oct 30 03:57:23.165298 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.165280    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:57:23.167914 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:23.167576    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode385d23f_6371_4f69_97b1_6a10303c048f.slice/crio-571ae192b8909614a7260b4ab6add5fc64778ca2eb4e4272773efcab9e70193c WatchSource:0}: Error finding container 571ae192b8909614a7260b4ab6add5fc64778ca2eb4e4272773efcab9e70193c: Status 404 returned error can't find the container with id 571ae192b8909614a7260b4ab6add5fc64778ca2eb4e4272773efcab9e70193c
Oct 30 03:57:23.197175 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:23.197156    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e672ce_27b7_4997_a887_b834e6a6797d.slice/crio-f4479c621470ed8e7bd2efe500ae1867cd75ab4ba570118cb724f77c5f5b180a WatchSource:0}: Error finding container f4479c621470ed8e7bd2efe500ae1867cd75ab4ba570118cb724f77c5f5b180a: Status 404 returned error can't find the container with id f4479c621470ed8e7bd2efe500ae1867cd75ab4ba570118cb724f77c5f5b180a
Oct 30 03:57:23.492585 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.492029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dec34b4-fb69-459e-b130-ca1716d23765-metrics-certs\") pod \"network-metrics-daemon-dpdr4\" (UID: \"4dec34b4-fb69-459e-b130-ca1716d23765\") " pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:23.492585 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.492293    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bad69b26-9de8-4c2b-b19c-8a96271f5c86-original-pull-secret\") pod \"global-pull-secret-syncer-cbfm8\" (UID: \"bad69b26-9de8-4c2b-b19c-8a96271f5c86\") " pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:23.492585 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:23.492369    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:57:23.492585 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:23.492440    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dec34b4-fb69-459e-b130-ca1716d23765-metrics-certs podName:4dec34b4-fb69-459e-b130-ca1716d23765 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:24.492421242 +0000 UTC m=+4.121940969 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4dec34b4-fb69-459e-b130-ca1716d23765-metrics-certs") pod "network-metrics-daemon-dpdr4" (UID: "4dec34b4-fb69-459e-b130-ca1716d23765") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:57:23.492585 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:23.492491    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 03:57:23.492585 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:23.492548    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bad69b26-9de8-4c2b-b19c-8a96271f5c86-original-pull-secret podName:bad69b26-9de8-4c2b-b19c-8a96271f5c86 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:24.492532817 +0000 UTC m=+4.122052554 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bad69b26-9de8-4c2b-b19c-8a96271f5c86-original-pull-secret") pod "global-pull-secret-syncer-cbfm8" (UID: "bad69b26-9de8-4c2b-b19c-8a96271f5c86") : object "kube-system"/"original-pull-secret" not registered
Oct 30 03:57:23.592753 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.592717    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwtgk\" (UniqueName: \"kubernetes.io/projected/d5e46634-9969-4d78-bb91-e598e950b1b0-kube-api-access-jwtgk\") pod \"network-check-target-9mktl\" (UID: \"d5e46634-9969-4d78-bb91-e598e950b1b0\") " pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:23.592917 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:23.592864    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 03:57:23.592917 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:23.592881    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 03:57:23.592917 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:23.592893    2578 projected.go:196] Error preparing data for projected volume kube-api-access-jwtgk for pod openshift-network-diagnostics/network-check-target-9mktl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:57:23.593083 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:23.592950    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e46634-9969-4d78-bb91-e598e950b1b0-kube-api-access-jwtgk podName:d5e46634-9969-4d78-bb91-e598e950b1b0 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:24.592932076 +0000 UTC m=+4.222451817 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jwtgk" (UniqueName: "kubernetes.io/projected/d5e46634-9969-4d78-bb91-e598e950b1b0-kube-api-access-jwtgk") pod "network-check-target-9mktl" (UID: "d5e46634-9969-4d78-bb91-e598e950b1b0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:57:23.851046 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.850910    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 03:52:22 +0000 UTC" deadline="2028-04-12 20:42:44.275231536 +0000 UTC"
Oct 30 03:57:23.851046 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.850961    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12736h45m20.4242745s"
Oct 30 03:57:23.855211 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.855182    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vnv9w" event={"ID":"e385d23f-6371-4f69-97b1-6a10303c048f","Type":"ContainerStarted","Data":"571ae192b8909614a7260b4ab6add5fc64778ca2eb4e4272773efcab9e70193c"}
Oct 30 03:57:23.857885 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.857858    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lrcg5" event={"ID":"455076cd-b484-44b6-8a1b-b4c3c0dccab8","Type":"ContainerStarted","Data":"77952d06a35c50393f61276f061dec66cdaa317609434651004514d570532709"}
Oct 30 03:57:23.863809 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.863730    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zl7qk" event={"ID":"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e","Type":"ContainerStarted","Data":"5523a203c9a2d75d7b7edb12c5dd54ef4c5676cc461ba54e3b38a768577c6ef0"}
Oct 30 03:57:23.867258 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.867224    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-244hd" event={"ID":"97de5e90-d514-4b8c-8917-2e9894978aaa","Type":"ContainerStarted","Data":"e99ebdc945a00b3970de14be54f3eabc56a7bfaed9413e043cc5311a161e1fb2"}
Oct 30 03:57:23.875593 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.875567    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ttsr6" event={"ID":"67ea77e7-74cc-4bc6-ae16-5a5a797f5097","Type":"ContainerStarted","Data":"1b399c10fd0f25666919e4b322b41999ca07b978930fc36c37ce9e191d6bef95"}
Oct 30 03:57:23.881100 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.881056    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8g477" event={"ID":"ade9bd13-6259-48aa-92b2-52f875b960e8","Type":"ContainerStarted","Data":"0121ea1bb37a9e8348390e8ef77e92d2f83e3f7208c8afd915debcfe1131184d"}
Oct 30 03:57:23.887715 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.887048    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-160.ec2.internal" event={"ID":"bc4b060adf5b640eabc59287108542f2","Type":"ContainerStarted","Data":"82d1a5d9fe6c5d4342a4dad2903f43d431c1b1f52cde6480ebfcc67c16cb3de6"}
Oct 30 03:57:23.891824 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.891799    2578 generic.go:356] "Generic (PLEG): container finished" podID="5bcce5d87346fa9cc82e9cdb4fa012bd" containerID="272526d88d54c2ab15fed2187134833e4c41d4abed1ebf882a8abc7e519cc400" exitCode=0
Oct 30 03:57:23.891906 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.891879    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal" event={"ID":"5bcce5d87346fa9cc82e9cdb4fa012bd","Type":"ContainerDied","Data":"272526d88d54c2ab15fed2187134833e4c41d4abed1ebf882a8abc7e519cc400"}
Oct 30 03:57:23.913587 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.913563    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nxvt" event={"ID":"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8","Type":"ContainerStarted","Data":"c0ddfb16511a887da3f3bd6c6bf63ae5d336361e54089180428a74ab9ca9f6f9"}
Oct 30 03:57:23.923271 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.923233    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4" event={"ID":"baf40c59-c04b-4c27-b080-6c55887c5c4c","Type":"ContainerStarted","Data":"78e7047f49c55034afe2ea6bda525cea7236ce655fbab85b43eb933b778551b1"}
Oct 30 03:57:23.931798 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.931773    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b88ss" event={"ID":"64f1a482-331f-4490-b03f-f41facd80cfe","Type":"ContainerStarted","Data":"65981a3a7a1f54c0ec3366558cc8b4a359ed404252cac9947b82dbcb41164bc8"}
Oct 30 03:57:23.939746 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.939714    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dgvb4" event={"ID":"30e672ce-27b7-4997-a887-b834e6a6797d","Type":"ContainerStarted","Data":"f4479c621470ed8e7bd2efe500ae1867cd75ab4ba570118cb724f77c5f5b180a"}
Oct 30 03:57:23.969961 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:23.969900    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-160.ec2.internal" podStartSLOduration=1.969882463 podStartE2EDuration="1.969882463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:23.927197825 +0000 UTC m=+3.556717570" watchObservedRunningTime="2026-10-30 03:57:23.969882463 +0000 UTC m=+3.599402197"
Oct 30 03:57:24.501077 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:24.501000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dec34b4-fb69-459e-b130-ca1716d23765-metrics-certs\") pod \"network-metrics-daemon-dpdr4\" (UID: \"4dec34b4-fb69-459e-b130-ca1716d23765\") " pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:24.501077 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:24.501045    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bad69b26-9de8-4c2b-b19c-8a96271f5c86-original-pull-secret\") pod \"global-pull-secret-syncer-cbfm8\" (UID: \"bad69b26-9de8-4c2b-b19c-8a96271f5c86\") " pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:24.501347 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:24.501174    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 03:57:24.501347 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:24.501251    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bad69b26-9de8-4c2b-b19c-8a96271f5c86-original-pull-secret podName:bad69b26-9de8-4c2b-b19c-8a96271f5c86 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:26.501218359 +0000 UTC m=+6.130738082 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bad69b26-9de8-4c2b-b19c-8a96271f5c86-original-pull-secret") pod "global-pull-secret-syncer-cbfm8" (UID: "bad69b26-9de8-4c2b-b19c-8a96271f5c86") : object "kube-system"/"original-pull-secret" not registered
Oct 30 03:57:24.501651 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:24.501634    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:57:24.501742 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:24.501686    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dec34b4-fb69-459e-b130-ca1716d23765-metrics-certs podName:4dec34b4-fb69-459e-b130-ca1716d23765 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:26.501671668 +0000 UTC m=+6.131191394 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4dec34b4-fb69-459e-b130-ca1716d23765-metrics-certs") pod "network-metrics-daemon-dpdr4" (UID: "4dec34b4-fb69-459e-b130-ca1716d23765") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:57:24.601969 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:24.601933    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwtgk\" (UniqueName: \"kubernetes.io/projected/d5e46634-9969-4d78-bb91-e598e950b1b0-kube-api-access-jwtgk\") pod \"network-check-target-9mktl\" (UID: \"d5e46634-9969-4d78-bb91-e598e950b1b0\") " pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:24.602161 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:24.602143    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 03:57:24.602223 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:24.602168    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 03:57:24.602223 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:24.602184    2578 projected.go:196] Error preparing data for projected volume kube-api-access-jwtgk for pod openshift-network-diagnostics/network-check-target-9mktl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:57:24.602340 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:24.602279    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e46634-9969-4d78-bb91-e598e950b1b0-kube-api-access-jwtgk podName:d5e46634-9969-4d78-bb91-e598e950b1b0 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:26.602255283 +0000 UTC m=+6.231775020 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jwtgk" (UniqueName: "kubernetes.io/projected/d5e46634-9969-4d78-bb91-e598e950b1b0-kube-api-access-jwtgk") pod "network-check-target-9mktl" (UID: "d5e46634-9969-4d78-bb91-e598e950b1b0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:57:24.833453 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:24.833371    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:24.833590 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:24.833499    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mktl" podUID="d5e46634-9969-4d78-bb91-e598e950b1b0"
Oct 30 03:57:24.833853 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:24.833830    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:24.833960 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:24.833915    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpdr4" podUID="4dec34b4-fb69-459e-b130-ca1716d23765"
Oct 30 03:57:24.834154 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:24.834138    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:24.834282 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:24.834261    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbfm8" podUID="bad69b26-9de8-4c2b-b19c-8a96271f5c86"
Oct 30 03:57:24.851218 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:24.851158    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 03:52:22 +0000 UTC" deadline="2028-03-26 13:38:58.381789802 +0000 UTC"
Oct 30 03:57:24.851708 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:24.851688    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12321h41m33.530110662s"
Oct 30 03:57:24.960339 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:24.960304    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal" event={"ID":"5bcce5d87346fa9cc82e9cdb4fa012bd","Type":"ContainerStarted","Data":"af8fa3e55af1d2904b369eb88159e9e7c9817e2e20ff32229cd6778163d3ffb3"}
Oct 30 03:57:25.975660 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:25.975486    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dgvb4" event={"ID":"30e672ce-27b7-4997-a887-b834e6a6797d","Type":"ContainerStarted","Data":"d2ff697079aee63dc22b2f07a49c242950d8b13e74a699ffd9151e2f4893c218"}
Oct 30 03:57:26.045178 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:26.045118    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal" podStartSLOduration=4.045099378 podStartE2EDuration="4.045099378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:25.000144633 +0000 UTC m=+4.629664395" watchObservedRunningTime="2026-10-30 03:57:26.045099378 +0000 UTC m=+5.674619127"
Oct 30 03:57:26.518584 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:26.518541    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dec34b4-fb69-459e-b130-ca1716d23765-metrics-certs\") pod \"network-metrics-daemon-dpdr4\" (UID: \"4dec34b4-fb69-459e-b130-ca1716d23765\") " pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:26.518761 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:26.518692    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:57:26.518819 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:26.518767    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dec34b4-fb69-459e-b130-ca1716d23765-metrics-certs podName:4dec34b4-fb69-459e-b130-ca1716d23765 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:30.518747305 +0000 UTC m=+10.148267064 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4dec34b4-fb69-459e-b130-ca1716d23765-metrics-certs") pod "network-metrics-daemon-dpdr4" (UID: "4dec34b4-fb69-459e-b130-ca1716d23765") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:57:26.519180 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:26.519110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bad69b26-9de8-4c2b-b19c-8a96271f5c86-original-pull-secret\") pod \"global-pull-secret-syncer-cbfm8\" (UID: \"bad69b26-9de8-4c2b-b19c-8a96271f5c86\") " pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:26.519299 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:26.519215    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 03:57:26.519299 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:26.519276    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bad69b26-9de8-4c2b-b19c-8a96271f5c86-original-pull-secret podName:bad69b26-9de8-4c2b-b19c-8a96271f5c86 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:30.519262468 +0000 UTC m=+10.148782204 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bad69b26-9de8-4c2b-b19c-8a96271f5c86-original-pull-secret") pod "global-pull-secret-syncer-cbfm8" (UID: "bad69b26-9de8-4c2b-b19c-8a96271f5c86") : object "kube-system"/"original-pull-secret" not registered
Oct 30 03:57:26.619872 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:26.619797    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwtgk\" (UniqueName: \"kubernetes.io/projected/d5e46634-9969-4d78-bb91-e598e950b1b0-kube-api-access-jwtgk\") pod \"network-check-target-9mktl\" (UID: \"d5e46634-9969-4d78-bb91-e598e950b1b0\") " pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:26.620027 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:26.619963    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 03:57:26.620027 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:26.619987    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 03:57:26.620027 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:26.619999    2578 projected.go:196] Error preparing data for projected volume kube-api-access-jwtgk for pod openshift-network-diagnostics/network-check-target-9mktl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:57:26.620180 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:26.620060    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e46634-9969-4d78-bb91-e598e950b1b0-kube-api-access-jwtgk podName:d5e46634-9969-4d78-bb91-e598e950b1b0 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:30.620040756 +0000 UTC m=+10.249560499 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jwtgk" (UniqueName: "kubernetes.io/projected/d5e46634-9969-4d78-bb91-e598e950b1b0-kube-api-access-jwtgk") pod "network-check-target-9mktl" (UID: "d5e46634-9969-4d78-bb91-e598e950b1b0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:57:26.833996 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:26.833961    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:26.834170 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:26.834093    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mktl" podUID="d5e46634-9969-4d78-bb91-e598e950b1b0"
Oct 30 03:57:26.834539 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:26.834512    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:26.834664 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:26.834645    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpdr4" podUID="4dec34b4-fb69-459e-b130-ca1716d23765"
Oct 30 03:57:26.834737 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:26.834702    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:26.834803 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:26.834778    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbfm8" podUID="bad69b26-9de8-4c2b-b19c-8a96271f5c86"
Oct 30 03:57:26.985629 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:26.985548    2578 generic.go:356] "Generic (PLEG): container finished" podID="4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8" containerID="d667734d25643e1d838954f2f2020f4c49c12d1e92ab9906b79d6e12edcfa53e" exitCode=0
Oct 30 03:57:26.986032 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:26.985641    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nxvt" event={"ID":"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8","Type":"ContainerDied","Data":"d667734d25643e1d838954f2f2020f4c49c12d1e92ab9906b79d6e12edcfa53e"}
Oct 30 03:57:26.993861 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:26.993832    2578 generic.go:356] "Generic (PLEG): container finished" podID="30e672ce-27b7-4997-a887-b834e6a6797d" containerID="d2ff697079aee63dc22b2f07a49c242950d8b13e74a699ffd9151e2f4893c218" exitCode=0
Oct 30 03:57:26.993962 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:26.993877    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dgvb4" event={"ID":"30e672ce-27b7-4997-a887-b834e6a6797d","Type":"ContainerDied","Data":"d2ff697079aee63dc22b2f07a49c242950d8b13e74a699ffd9151e2f4893c218"}
Oct 30 03:57:28.001301 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:28.001266    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dgvb4" event={"ID":"30e672ce-27b7-4997-a887-b834e6a6797d","Type":"ContainerStarted","Data":"bfad6cc6053f55deac3a8819b93a28b0edcf1e29755416985bb335a5080b0974"}
Oct 30 03:57:28.001752 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:28.001353    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dgvb4" event={"ID":"30e672ce-27b7-4997-a887-b834e6a6797d","Type":"ContainerStarted","Data":"ce7f1ae7be5f13d4e21fb75a3accd38a6d5de87bb963ae7a340263464bc6e041"}
Oct 30 03:57:28.006983 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:28.006952    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vnv9w" event={"ID":"e385d23f-6371-4f69-97b1-6a10303c048f","Type":"ContainerStarted","Data":"8eb04dc9edb523b799aa0f3bdf908e8b5aaae640354e8f908d523c92a0bee52f"}
Oct 30 03:57:28.097415 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:28.097345    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dgvb4" podStartSLOduration=3.536202968 podStartE2EDuration="6.097325799s" totalImagesPullingTime="2.48132749s" totalInitContainerRuntime="79.795341ms" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:28.047280367 +0000 UTC m=+7.676800113" watchObservedRunningTime="2026-10-30 03:57:28.097325799 +0000 UTC m=+7.726845544"
Oct 30 03:57:28.833489 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:28.833466    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:28.833603 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:28.833563    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mktl" podUID="d5e46634-9969-4d78-bb91-e598e950b1b0"
Oct 30 03:57:28.833670 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:28.833616    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:28.833719 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:28.833673    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpdr4" podUID="4dec34b4-fb69-459e-b130-ca1716d23765"
Oct 30 03:57:28.833772 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:28.833720    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:28.833909 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:28.833785    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbfm8" podUID="bad69b26-9de8-4c2b-b19c-8a96271f5c86"
Oct 30 03:57:29.015171 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:29.014880    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4" event={"ID":"baf40c59-c04b-4c27-b080-6c55887c5c4c","Type":"ContainerStarted","Data":"79f32cc63a0afc7a0fc21588eb173c9b8c04eaef79b56b1f047cfcfda2533f26"}
Oct 30 03:57:29.020145 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:29.020118    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lrcg5" event={"ID":"455076cd-b484-44b6-8a1b-b4c3c0dccab8","Type":"ContainerStarted","Data":"a59d1a23fd53b45a1e92e9c22e836fc38c9e66b9c88be604745fa78a4fc3be2c"}
Oct 30 03:57:29.058691 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:29.058634    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vnv9w" podStartSLOduration=4.190189016 podStartE2EDuration="8.058616292s" totalImagesPullingTime="3.868427276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:28.098469628 +0000 UTC m=+7.727989375" watchObservedRunningTime="2026-10-30 03:57:29.058616292 +0000 UTC m=+8.688136035"
Oct 30 03:57:30.010317 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:30.010280    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vnv9w"
Oct 30 03:57:30.010716 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:30.010693    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vnv9w"
Oct 30 03:57:30.054227 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:30.054155    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lrcg5" podStartSLOduration=3.996226963 podStartE2EDuration="9.054138313s" totalImagesPullingTime="5.05791135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:29.058863429 +0000 UTC m=+8.688383173" watchObservedRunningTime="2026-10-30 03:57:30.054138313 +0000 UTC m=+9.683658058"
Oct 30 03:57:30.415733 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:30.415667    2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 03:57:30.554639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:30.553872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dec34b4-fb69-459e-b130-ca1716d23765-metrics-certs\") pod \"network-metrics-daemon-dpdr4\" (UID: \"4dec34b4-fb69-459e-b130-ca1716d23765\") " pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:30.554639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:30.553913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bad69b26-9de8-4c2b-b19c-8a96271f5c86-original-pull-secret\") pod \"global-pull-secret-syncer-cbfm8\" (UID: \"bad69b26-9de8-4c2b-b19c-8a96271f5c86\") " pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:30.554639 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:30.554042    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 03:57:30.554639 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:30.554099    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bad69b26-9de8-4c2b-b19c-8a96271f5c86-original-pull-secret podName:bad69b26-9de8-4c2b-b19c-8a96271f5c86 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:38.554080567 +0000 UTC m=+18.183600306 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bad69b26-9de8-4c2b-b19c-8a96271f5c86-original-pull-secret") pod "global-pull-secret-syncer-cbfm8" (UID: "bad69b26-9de8-4c2b-b19c-8a96271f5c86") : object "kube-system"/"original-pull-secret" not registered
Oct 30 03:57:30.554639 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:30.554521    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:57:30.554639 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:30.554564    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dec34b4-fb69-459e-b130-ca1716d23765-metrics-certs podName:4dec34b4-fb69-459e-b130-ca1716d23765 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:38.554550416 +0000 UTC m=+18.184070152 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4dec34b4-fb69-459e-b130-ca1716d23765-metrics-certs") pod "network-metrics-daemon-dpdr4" (UID: "4dec34b4-fb69-459e-b130-ca1716d23765") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:57:30.655213 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:30.655169    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwtgk\" (UniqueName: \"kubernetes.io/projected/d5e46634-9969-4d78-bb91-e598e950b1b0-kube-api-access-jwtgk\") pod \"network-check-target-9mktl\" (UID: \"d5e46634-9969-4d78-bb91-e598e950b1b0\") " pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:30.655455 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:30.655401    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 03:57:30.655455 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:30.655431    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 03:57:30.655455 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:30.655445    2578 projected.go:196] Error preparing data for projected volume kube-api-access-jwtgk for pod openshift-network-diagnostics/network-check-target-9mktl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:57:30.655663 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:30.655512    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e46634-9969-4d78-bb91-e598e950b1b0-kube-api-access-jwtgk podName:d5e46634-9969-4d78-bb91-e598e950b1b0 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:38.655491822 +0000 UTC m=+18.285011545 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jwtgk" (UniqueName: "kubernetes.io/projected/d5e46634-9969-4d78-bb91-e598e950b1b0-kube-api-access-jwtgk") pod "network-check-target-9mktl" (UID: "d5e46634-9969-4d78-bb91-e598e950b1b0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:57:30.834368 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:30.833078    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:30.834368 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:30.833434    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mktl" podUID="d5e46634-9969-4d78-bb91-e598e950b1b0"
Oct 30 03:57:30.834368 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:30.833828    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:30.834368 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:30.833919    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpdr4" podUID="4dec34b4-fb69-459e-b130-ca1716d23765"
Oct 30 03:57:30.834368 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:30.833979    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:30.834368 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:30.834038    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbfm8" podUID="bad69b26-9de8-4c2b-b19c-8a96271f5c86"
Oct 30 03:57:30.855899 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:30.855783    2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T03:57:30.415689549Z","UUID":"de8f594f-ce96-4653-8427-81643300aa40","Handler":null,"Name":"","Endpoint":""}
Oct 30 03:57:30.857660 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:30.857419    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 03:57:30.857660 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:30.857448    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 03:57:31.031507 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:31.031455    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4" event={"ID":"baf40c59-c04b-4c27-b080-6c55887c5c4c","Type":"ContainerStarted","Data":"8f5ad718f38bb3267b5e3b4c20c13e8b114cce5d481ea05f9ea706b4555f4bd2"}
Oct 30 03:57:31.040652 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:31.040631    2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 03:57:31.041575 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:31.041549    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zl7qk" event={"ID":"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e","Type":"ContainerStarted","Data":"f120ef71accd469a7ae19b1d36e2195d947762e8bb25914d6374d4574f5d2763"}
Oct 30 03:57:32.044986 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:32.044565    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8g477" event={"ID":"ade9bd13-6259-48aa-92b2-52f875b960e8","Type":"ContainerStarted","Data":"e9646d06289ac871834b0651551cb448a56dae053bb6b3534edb59ad776cac58"}
Oct 30 03:57:32.046692 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:32.046600    2578 generic.go:356] "Generic (PLEG): container finished" podID="4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8" containerID="1af63fe35fccea7a7c1ada89683a1c6d92a961ac01113f8ada8653bf9e6321bb" exitCode=0
Oct 30 03:57:32.046778 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:32.046685    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nxvt" event={"ID":"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8","Type":"ContainerDied","Data":"1af63fe35fccea7a7c1ada89683a1c6d92a961ac01113f8ada8653bf9e6321bb"}
Oct 30 03:57:32.050326 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:32.050301    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4" event={"ID":"baf40c59-c04b-4c27-b080-6c55887c5c4c","Type":"ContainerStarted","Data":"e9d8ce09da426744f7e530f80d1f87beffe6cdb196c37473a95ac11cd9cff000"}
Oct 30 03:57:32.082681 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:32.082633    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-8g477" podStartSLOduration=4.075476066 podStartE2EDuration="12.082620809s" totalImagesPullingTime="8.007144743s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:32.08232687 +0000 UTC m=+11.711846615" watchObservedRunningTime="2026-10-30 03:57:32.082620809 +0000 UTC m=+11.712140552"
Oct 30 03:57:32.083218 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:32.083179    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-zl7qk" podStartSLOduration=3.54858014 podStartE2EDuration="11.083172272s" totalImagesPullingTime="7.534592132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:31.090020338 +0000 UTC m=+10.719540079" watchObservedRunningTime="2026-10-30 03:57:32.083172272 +0000 UTC m=+11.712692015"
Oct 30 03:57:32.125364 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:32.125314    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4" podStartSLOduration=3.154951093 podStartE2EDuration="11.12529731s" totalImagesPullingTime="7.970346217s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:32.124485583 +0000 UTC m=+11.754005327" watchObservedRunningTime="2026-10-30 03:57:32.12529731 +0000 UTC m=+11.754817054"
Oct 30 03:57:32.832046 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:32.832019    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:32.832165 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:32.832096    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:32.832221 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:32.832212    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:32.832289 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:32.832208    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpdr4" podUID="4dec34b4-fb69-459e-b130-ca1716d23765"
Oct 30 03:57:32.832356 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:32.832299    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbfm8" podUID="bad69b26-9de8-4c2b-b19c-8a96271f5c86"
Oct 30 03:57:32.832407 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:32.832367    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mktl" podUID="d5e46634-9969-4d78-bb91-e598e950b1b0"
Oct 30 03:57:33.053880 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:33.053847    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ttsr6" event={"ID":"67ea77e7-74cc-4bc6-ae16-5a5a797f5097","Type":"ContainerStarted","Data":"f4f2837a0efaabaac78bd7a2459942893a840a1df09632f6dcc91397f19d6480"}
Oct 30 03:57:33.056967 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:33.056935    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nxvt" event={"ID":"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8","Type":"ContainerStarted","Data":"f36e067f9b4019ff1a3a82fe38b077e7b124f4710d92241f4ccec47e218bf92f"}
Oct 30 03:57:33.091755 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:33.091717    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ttsr6" podStartSLOduration=4.975474684 podStartE2EDuration="13.091705481s" totalImagesPullingTime="8.116230797s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:33.091535139 +0000 UTC m=+12.721054884" watchObservedRunningTime="2026-10-30 03:57:33.091705481 +0000 UTC m=+12.721225225"
Oct 30 03:57:34.061755 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:34.061553    2578 generic.go:356] "Generic (PLEG): container finished" podID="4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8" containerID="f36e067f9b4019ff1a3a82fe38b077e7b124f4710d92241f4ccec47e218bf92f" exitCode=0
Oct 30 03:57:34.062282 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:34.061645    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nxvt" event={"ID":"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8","Type":"ContainerDied","Data":"f36e067f9b4019ff1a3a82fe38b077e7b124f4710d92241f4ccec47e218bf92f"}
Oct 30 03:57:34.063786 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:34.063767    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b88ss" event={"ID":"64f1a482-331f-4490-b03f-f41facd80cfe","Type":"ContainerStarted","Data":"d5afeb9574c8bf871c8ef69b4e8fcda30282ae6da6d74440737074d29b8f239c"}
Oct 30 03:57:34.229363 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:34.229313    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-b88ss" podStartSLOduration=4.078014276 podStartE2EDuration="14.229300719s" totalImagesPullingTime="10.151286443s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:34.228940614 +0000 UTC m=+13.858460359" watchObservedRunningTime="2026-10-30 03:57:34.229300719 +0000 UTC m=+13.858820462"
Oct 30 03:57:34.831355 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:34.831332    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:34.831473 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:34.831366    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:34.831473 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:34.831366    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:34.831473 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:34.831449    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mktl" podUID="d5e46634-9969-4d78-bb91-e598e950b1b0"
Oct 30 03:57:34.831870 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:34.831845    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpdr4" podUID="4dec34b4-fb69-459e-b130-ca1716d23765"
Oct 30 03:57:34.831970 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:34.831935    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbfm8" podUID="bad69b26-9de8-4c2b-b19c-8a96271f5c86"
Oct 30 03:57:35.068478 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:35.068397    2578 generic.go:356] "Generic (PLEG): container finished" podID="4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8" containerID="fca7af361ec57381ea41e64351e6a33fb35eb6a2da611435fab3c75245dc4350" exitCode=0
Oct 30 03:57:35.068478 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:35.068431    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nxvt" event={"ID":"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8","Type":"ContainerDied","Data":"fca7af361ec57381ea41e64351e6a33fb35eb6a2da611435fab3c75245dc4350"}
Oct 30 03:57:36.832038 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:36.832013    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:36.832385 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:36.832020    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:36.832385 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:36.832146    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:36.832385 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:36.832153    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mktl" podUID="d5e46634-9969-4d78-bb91-e598e950b1b0"
Oct 30 03:57:36.832385 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:36.832284    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpdr4" podUID="4dec34b4-fb69-459e-b130-ca1716d23765"
Oct 30 03:57:36.832385 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:36.832326    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbfm8" podUID="bad69b26-9de8-4c2b-b19c-8a96271f5c86"
Oct 30 03:57:37.074457 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:37.074428    2578 generic.go:356] "Generic (PLEG): container finished" podID="4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8" containerID="d6c5b21006b70ee9735297284090399a087d1a2dc8fd743a61bb14bc971a795c" exitCode=0
Oct 30 03:57:37.074569 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:37.074494    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nxvt" event={"ID":"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8","Type":"ContainerDied","Data":"d6c5b21006b70ee9735297284090399a087d1a2dc8fd743a61bb14bc971a795c"}
Oct 30 03:57:38.079642 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:38.079411    2578 generic.go:356] "Generic (PLEG): container finished" podID="4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8" containerID="f449f312f714a685ea8bd0cdb6a12904911d9d08af814de7b93e4085bbebbac6" exitCode=0
Oct 30 03:57:38.080286 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:38.079504    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nxvt" event={"ID":"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8","Type":"ContainerDied","Data":"f449f312f714a685ea8bd0cdb6a12904911d9d08af814de7b93e4085bbebbac6"}
Oct 30 03:57:38.082617 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:38.082597    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 03:57:38.082939 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:38.082920    2578 generic.go:356] "Generic (PLEG): container finished" podID="97de5e90-d514-4b8c-8917-2e9894978aaa" containerID="5714c57cd1973935027280deb7dededf26ab4da87ba0ae006351458ecb0f6bc8" exitCode=1
Oct 30 03:57:38.082996 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:38.082956    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-244hd" event={"ID":"97de5e90-d514-4b8c-8917-2e9894978aaa","Type":"ContainerStarted","Data":"7650c9226ae0b415b9be1dc431b14823f4f1bbdcc41c2e99cfa50a78f40ec0d6"}
Oct 30 03:57:38.083028 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:38.083007    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-244hd" event={"ID":"97de5e90-d514-4b8c-8917-2e9894978aaa","Type":"ContainerStarted","Data":"2e2be6d8210df691fccee9622198de5a371263baf6f3018bc875c23dc2450911"}
Oct 30 03:57:38.083057 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:38.083040    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-244hd" event={"ID":"97de5e90-d514-4b8c-8917-2e9894978aaa","Type":"ContainerStarted","Data":"d13b0df1654579b9613965ae23cd31de0730933ef0c68d426d5137a84c33d835"}
Oct 30 03:57:38.083088 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:38.083067    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-244hd" event={"ID":"97de5e90-d514-4b8c-8917-2e9894978aaa","Type":"ContainerStarted","Data":"ebf20351b21d0e143344cd8b67b484f330000f53272e32ff21151293a28c3a26"}
Oct 30 03:57:38.083119 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:38.083093    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-244hd" event={"ID":"97de5e90-d514-4b8c-8917-2e9894978aaa","Type":"ContainerDied","Data":"5714c57cd1973935027280deb7dededf26ab4da87ba0ae006351458ecb0f6bc8"}
Oct 30 03:57:38.083147 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:38.083121    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-244hd" event={"ID":"97de5e90-d514-4b8c-8917-2e9894978aaa","Type":"ContainerStarted","Data":"4768a355b0410ad026aace4d0794ce685d363f27b478118801f8c3cd7a8f51b9"}
Oct 30 03:57:38.616362 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:38.616337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dec34b4-fb69-459e-b130-ca1716d23765-metrics-certs\") pod \"network-metrics-daemon-dpdr4\" (UID: \"4dec34b4-fb69-459e-b130-ca1716d23765\") " pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:38.616503 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:38.616366    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bad69b26-9de8-4c2b-b19c-8a96271f5c86-original-pull-secret\") pod \"global-pull-secret-syncer-cbfm8\" (UID: \"bad69b26-9de8-4c2b-b19c-8a96271f5c86\") " pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:38.616503 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:38.616474    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 03:57:38.616503 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:38.616477    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:57:38.616601 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:38.616518    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bad69b26-9de8-4c2b-b19c-8a96271f5c86-original-pull-secret podName:bad69b26-9de8-4c2b-b19c-8a96271f5c86 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:54.616506646 +0000 UTC m=+34.246026381 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bad69b26-9de8-4c2b-b19c-8a96271f5c86-original-pull-secret") pod "global-pull-secret-syncer-cbfm8" (UID: "bad69b26-9de8-4c2b-b19c-8a96271f5c86") : object "kube-system"/"original-pull-secret" not registered
Oct 30 03:57:38.616601 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:38.616532    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dec34b4-fb69-459e-b130-ca1716d23765-metrics-certs podName:4dec34b4-fb69-459e-b130-ca1716d23765 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:54.616525363 +0000 UTC m=+34.246045085 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4dec34b4-fb69-459e-b130-ca1716d23765-metrics-certs") pod "network-metrics-daemon-dpdr4" (UID: "4dec34b4-fb69-459e-b130-ca1716d23765") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:57:38.716960 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:38.716932    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwtgk\" (UniqueName: \"kubernetes.io/projected/d5e46634-9969-4d78-bb91-e598e950b1b0-kube-api-access-jwtgk\") pod \"network-check-target-9mktl\" (UID: \"d5e46634-9969-4d78-bb91-e598e950b1b0\") " pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:38.717096 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:38.717077    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 03:57:38.717131 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:38.717103    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 03:57:38.717131 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:38.717114    2578 projected.go:196] Error preparing data for projected volume kube-api-access-jwtgk for pod openshift-network-diagnostics/network-check-target-9mktl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:57:38.717202 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:38.717156    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e46634-9969-4d78-bb91-e598e950b1b0-kube-api-access-jwtgk podName:d5e46634-9969-4d78-bb91-e598e950b1b0 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:54.717144041 +0000 UTC m=+34.346663763 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jwtgk" (UniqueName: "kubernetes.io/projected/d5e46634-9969-4d78-bb91-e598e950b1b0-kube-api-access-jwtgk") pod "network-check-target-9mktl" (UID: "d5e46634-9969-4d78-bb91-e598e950b1b0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:57:38.831637 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:38.831609    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:38.831637 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:38.831623    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:38.831781 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:38.831709    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpdr4" podUID="4dec34b4-fb69-459e-b130-ca1716d23765"
Oct 30 03:57:38.831828 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:38.831799    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbfm8" podUID="bad69b26-9de8-4c2b-b19c-8a96271f5c86"
Oct 30 03:57:38.831912 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:38.831872    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:38.831983 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:38.831959    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mktl" podUID="d5e46634-9969-4d78-bb91-e598e950b1b0"
Oct 30 03:57:39.088132 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:39.088053    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nxvt" event={"ID":"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8","Type":"ContainerStarted","Data":"c7c66fa94835c0c684f54c46896b17ada4a02fbce4d944df411c0bec80b80240"}
Oct 30 03:57:39.157281 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:39.157215    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9nxvt" podStartSLOduration=7.733072151 podStartE2EDuration="18.157203157s" totalImagesPullingTime="10.424131006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:39.145005264 +0000 UTC m=+18.774525007" watchObservedRunningTime="2026-10-30 03:57:39.157203157 +0000 UTC m=+18.786722901"
Oct 30 03:57:40.092341 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:40.092316    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 03:57:40.092815 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:40.092748    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-244hd" event={"ID":"97de5e90-d514-4b8c-8917-2e9894978aaa","Type":"ContainerStarted","Data":"829d8cbe51096f4d13803e45cd563a6eb615394a802984c10fb2e693790604bd"}
Oct 30 03:57:40.832657 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:40.832625    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:40.832819 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:40.832704    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mktl" podUID="d5e46634-9969-4d78-bb91-e598e950b1b0"
Oct 30 03:57:40.832819 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:40.832782    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:40.832941 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:40.832860    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:40.832992 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:40.832935    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpdr4" podUID="4dec34b4-fb69-459e-b130-ca1716d23765"
Oct 30 03:57:40.832992 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:40.832928    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbfm8" podUID="bad69b26-9de8-4c2b-b19c-8a96271f5c86"
Oct 30 03:57:42.099774 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:42.099542    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 03:57:42.100207 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:42.100025    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-244hd" event={"ID":"97de5e90-d514-4b8c-8917-2e9894978aaa","Type":"ContainerStarted","Data":"9fe5ea9d1eca8120585a56d964f646442711865d42083dd218af1bf3f0e9ba16"}
Oct 30 03:57:42.100317 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:42.100297    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:42.100377 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:42.100366    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:42.100487 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:42.100471    2578 scope.go:122] "RemoveContainer" containerID="5714c57cd1973935027280deb7dededf26ab4da87ba0ae006351458ecb0f6bc8"
Oct 30 03:57:42.116411 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:42.116392    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:42.832044 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:42.832010    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:42.832044 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:42.832048    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:42.832272 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:42.832116    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbfm8" podUID="bad69b26-9de8-4c2b-b19c-8a96271f5c86"
Oct 30 03:57:42.832272 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:42.832018    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:42.832272 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:42.832187    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mktl" podUID="d5e46634-9969-4d78-bb91-e598e950b1b0"
Oct 30 03:57:42.832439 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:42.832292    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpdr4" podUID="4dec34b4-fb69-459e-b130-ca1716d23765"
Oct 30 03:57:43.105889 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:43.105825    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 03:57:43.106324 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:43.106166    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-244hd" event={"ID":"97de5e90-d514-4b8c-8917-2e9894978aaa","Type":"ContainerStarted","Data":"e91a6764cdc6c0aee497a8825b2e63232a4125742a9faf9421c7c427e79857b5"}
Oct 30 03:57:43.106405 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:43.106389    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:43.120695 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:43.120675    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:57:43.195548 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:43.195498    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-244hd" podStartSLOduration=9.27990732 podStartE2EDuration="23.195486936s" totalImagesPullingTime="13.915579616s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:43.182933414 +0000 UTC m=+22.812453161" watchObservedRunningTime="2026-10-30 03:57:43.195486936 +0000 UTC m=+22.825006680"
Oct 30 03:57:43.747755 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:43.747688    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dpdr4"]
Oct 30 03:57:43.747888 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:43.747875    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:43.747996 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:43.747976    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpdr4" podUID="4dec34b4-fb69-459e-b130-ca1716d23765"
Oct 30 03:57:43.751952 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:43.751926    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9mktl"]
Oct 30 03:57:43.752054 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:43.751977    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cbfm8"]
Oct 30 03:57:43.752054 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:43.752039    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:43.752122 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:43.752104    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbfm8" podUID="bad69b26-9de8-4c2b-b19c-8a96271f5c86"
Oct 30 03:57:43.752165 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:43.752152    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:43.752204 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:43.752192    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mktl" podUID="d5e46634-9969-4d78-bb91-e598e950b1b0"
Oct 30 03:57:44.108190 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:44.108112    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vnv9w"
Oct 30 03:57:44.108636 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:44.108223    2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 03:57:44.108636 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:44.108587    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vnv9w"
Oct 30 03:57:44.834918 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:44.834885    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:44.835174 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:44.834997    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbfm8" podUID="bad69b26-9de8-4c2b-b19c-8a96271f5c86"
Oct 30 03:57:45.832255 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:45.832220    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:45.832589 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:45.832339    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:45.832589 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:45.832419    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpdr4" podUID="4dec34b4-fb69-459e-b130-ca1716d23765"
Oct 30 03:57:45.832589 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:45.832405    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mktl" podUID="d5e46634-9969-4d78-bb91-e598e950b1b0"
Oct 30 03:57:46.834298 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:46.834105    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:46.834674 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:46.834369    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbfm8" podUID="bad69b26-9de8-4c2b-b19c-8a96271f5c86"
Oct 30 03:57:47.832222 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:47.832171    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:47.832222 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:47.832203    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:47.832499 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:47.832300    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mktl" podUID="d5e46634-9969-4d78-bb91-e598e950b1b0"
Oct 30 03:57:47.832499 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:47.832417    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpdr4" podUID="4dec34b4-fb69-459e-b130-ca1716d23765"
Oct 30 03:57:48.836544 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:48.836517    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:48.836894 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:48.836616    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cbfm8" podUID="bad69b26-9de8-4c2b-b19c-8a96271f5c86"
Oct 30 03:57:49.832346 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:49.832315    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:49.832480 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:49.832415    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mktl" podUID="d5e46634-9969-4d78-bb91-e598e950b1b0"
Oct 30 03:57:49.832480 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:49.832467    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:49.832595 ip-10-0-138-160 kubenswrapper[2578]: E1030 03:57:49.832577    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dpdr4" podUID="4dec34b4-fb69-459e-b130-ca1716d23765"
Oct 30 03:57:50.656574 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.656550    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeReady"
Oct 30 03:57:50.657008 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.656657    2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 03:57:50.720879 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.720850    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8qhbh"]
Oct 30 03:57:50.762108 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.762084    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2ldpw"]
Oct 30 03:57:50.762284 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.762200    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:57:50.765522 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.765500    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8qhbh"]
Oct 30 03:57:50.765650 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.765537    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2ldpw"]
Oct 30 03:57:50.765650 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.765558    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mctfq"]
Oct 30 03:57:50.765650 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.765619    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2ldpw"
Oct 30 03:57:50.778466 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.778445    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mctfq"
Oct 30 03:57:50.783932 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.783878    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 03:57:50.783988 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.783980    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-846td\""
Oct 30 03:57:50.784050 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.784039    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 03:57:50.784085 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.784069    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 03:57:50.784118 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.784112    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 03:57:50.795474 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.795454    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 03:57:50.795632 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.795619    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 03:57:50.796488 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.796471    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mctfq"]
Oct 30 03:57:50.831804 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.831788    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:50.884528 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.884510    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 03:57:50.884639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.884518    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 03:57:50.884639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.884565    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 03:57:50.884639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.884625    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b2qtm\""
Oct 30 03:57:50.884765 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.884688    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hmk7j\""
Oct 30 03:57:50.884765 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.884750    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 03:57:50.884851 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.884804    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j9mnz\""
Oct 30 03:57:50.906289 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.906269    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6755b1f-578f-4e75-86df-be3db7533610-metrics-tls\") pod \"dns-default-mctfq\" (UID: \"a6755b1f-578f-4e75-86df-be3db7533610\") " pod="openshift-dns/dns-default-mctfq"
Oct 30 03:57:50.906376 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.906296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a6755b1f-578f-4e75-86df-be3db7533610-config-volume\") pod \"dns-default-mctfq\" (UID: \"a6755b1f-578f-4e75-86df-be3db7533610\") " pod="openshift-dns/dns-default-mctfq"
Oct 30 03:57:50.906492 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.906472    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-crio-socket\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:57:50.906529 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.906512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:57:50.906569 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.906558    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bee7738d-afa2-431d-ba81-2d14f142902c-cert\") pod \"ingress-canary-2ldpw\" (UID: \"bee7738d-afa2-431d-ba81-2d14f142902c\") " pod="openshift-ingress-canary/ingress-canary-2ldpw"
Oct 30 03:57:50.906633 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.906582    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h268\" (UniqueName: \"kubernetes.io/projected/bee7738d-afa2-431d-ba81-2d14f142902c-kube-api-access-6h268\") pod \"ingress-canary-2ldpw\" (UID: \"bee7738d-afa2-431d-ba81-2d14f142902c\") " pod="openshift-ingress-canary/ingress-canary-2ldpw"
Oct 30 03:57:50.906633 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.906611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:57:50.906727 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.906650    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-data-volume\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:57:50.906727 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.906708    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt7mf\" (UniqueName: \"kubernetes.io/projected/a6755b1f-578f-4e75-86df-be3db7533610-kube-api-access-dt7mf\") pod \"dns-default-mctfq\" (UID: \"a6755b1f-578f-4e75-86df-be3db7533610\") " pod="openshift-dns/dns-default-mctfq"
Oct 30 03:57:50.906797 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.906749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6755b1f-578f-4e75-86df-be3db7533610-tmp-dir\") pod \"dns-default-mctfq\" (UID: \"a6755b1f-578f-4e75-86df-be3db7533610\") " pod="openshift-dns/dns-default-mctfq"
Oct 30 03:57:50.906797 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:50.906775    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6msml\" (UniqueName: \"kubernetes.io/projected/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-kube-api-access-6msml\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:57:51.007651 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.007625    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-crio-socket\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:57:51.007758 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.007657    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:57:51.007758 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.007680    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bee7738d-afa2-431d-ba81-2d14f142902c-cert\") pod \"ingress-canary-2ldpw\" (UID: \"bee7738d-afa2-431d-ba81-2d14f142902c\") " pod="openshift-ingress-canary/ingress-canary-2ldpw"
Oct 30 03:57:51.007758 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.007702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h268\" (UniqueName: \"kubernetes.io/projected/bee7738d-afa2-431d-ba81-2d14f142902c-kube-api-access-6h268\") pod \"ingress-canary-2ldpw\" (UID: \"bee7738d-afa2-431d-ba81-2d14f142902c\") " pod="openshift-ingress-canary/ingress-canary-2ldpw"
Oct 30 03:57:51.007758 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.007729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:57:51.007949 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.007816    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-crio-socket\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:57:51.007949 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.007829    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-data-volume\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:57:51.007949 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.007871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt7mf\" (UniqueName: \"kubernetes.io/projected/a6755b1f-578f-4e75-86df-be3db7533610-kube-api-access-dt7mf\") pod \"dns-default-mctfq\" (UID: \"a6755b1f-578f-4e75-86df-be3db7533610\") " pod="openshift-dns/dns-default-mctfq"
Oct 30 03:57:51.007949 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.007908    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6755b1f-578f-4e75-86df-be3db7533610-tmp-dir\") pod \"dns-default-mctfq\" (UID: \"a6755b1f-578f-4e75-86df-be3db7533610\") " pod="openshift-dns/dns-default-mctfq"
Oct 30 03:57:51.007949 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.007936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6msml\" (UniqueName: \"kubernetes.io/projected/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-kube-api-access-6msml\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:57:51.008186 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.007979    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6755b1f-578f-4e75-86df-be3db7533610-metrics-tls\") pod \"dns-default-mctfq\" (UID: \"a6755b1f-578f-4e75-86df-be3db7533610\") " pod="openshift-dns/dns-default-mctfq"
Oct 30 03:57:51.008186 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.008003    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a6755b1f-578f-4e75-86df-be3db7533610-config-volume\") pod \"dns-default-mctfq\" (UID: \"a6755b1f-578f-4e75-86df-be3db7533610\") " pod="openshift-dns/dns-default-mctfq"
Oct 30 03:57:51.008306 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.008232    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-data-volume\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:57:51.008513 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.008493    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6755b1f-578f-4e75-86df-be3db7533610-tmp-dir\") pod \"dns-default-mctfq\" (UID: \"a6755b1f-578f-4e75-86df-be3db7533610\") " pod="openshift-dns/dns-default-mctfq"
Oct 30 03:57:51.008639 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.008620    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:57:51.008700 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.008681    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a6755b1f-578f-4e75-86df-be3db7533610-config-volume\") pod \"dns-default-mctfq\" (UID: \"a6755b1f-578f-4e75-86df-be3db7533610\") " pod="openshift-dns/dns-default-mctfq"
Oct 30 03:57:51.011686 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.011664    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:57:51.011686 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.011682    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6755b1f-578f-4e75-86df-be3db7533610-metrics-tls\") pod \"dns-default-mctfq\" (UID: \"a6755b1f-578f-4e75-86df-be3db7533610\") " pod="openshift-dns/dns-default-mctfq"
Oct 30 03:57:51.011805 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.011723    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bee7738d-afa2-431d-ba81-2d14f142902c-cert\") pod \"ingress-canary-2ldpw\" (UID: \"bee7738d-afa2-431d-ba81-2d14f142902c\") " pod="openshift-ingress-canary/ingress-canary-2ldpw"
Oct 30 03:57:51.031589 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.031562    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h268\" (UniqueName: \"kubernetes.io/projected/bee7738d-afa2-431d-ba81-2d14f142902c-kube-api-access-6h268\") pod \"ingress-canary-2ldpw\" (UID: \"bee7738d-afa2-431d-ba81-2d14f142902c\") " pod="openshift-ingress-canary/ingress-canary-2ldpw"
Oct 30 03:57:51.031769 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.031586    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt7mf\" (UniqueName: \"kubernetes.io/projected/a6755b1f-578f-4e75-86df-be3db7533610-kube-api-access-dt7mf\") pod \"dns-default-mctfq\" (UID: \"a6755b1f-578f-4e75-86df-be3db7533610\") " pod="openshift-dns/dns-default-mctfq"
Oct 30 03:57:51.031769 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.031626    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6msml\" (UniqueName: \"kubernetes.io/projected/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-kube-api-access-6msml\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:57:51.130750 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.130724    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:57:51.168608 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.168582    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2ldpw"
Oct 30 03:57:51.196138 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.196116    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mctfq"
Oct 30 03:57:51.502046 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.502016    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2ldpw"]
Oct 30 03:57:51.507700 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:51.507675    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee7738d_afa2_431d_ba81_2d14f142902c.slice/crio-822a1f9a7cf577e4192c455edd2192ac1dcb180d38395f5438a3ff7757c4e19c WatchSource:0}: Error finding container 822a1f9a7cf577e4192c455edd2192ac1dcb180d38395f5438a3ff7757c4e19c: Status 404 returned error can't find the container with id 822a1f9a7cf577e4192c455edd2192ac1dcb180d38395f5438a3ff7757c4e19c
Oct 30 03:57:51.517408 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.517331    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mctfq"]
Oct 30 03:57:51.519527 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:51.519501    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6755b1f_578f_4e75_86df_be3db7533610.slice/crio-e73929c55e45a035187a471a0bd4973ec6348478df05c7b93f9d7b34d69b9327 WatchSource:0}: Error finding container e73929c55e45a035187a471a0bd4973ec6348478df05c7b93f9d7b34d69b9327: Status 404 returned error can't find the container with id e73929c55e45a035187a471a0bd4973ec6348478df05c7b93f9d7b34d69b9327
Oct 30 03:57:51.520875 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.520854    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8qhbh"]
Oct 30 03:57:51.525270 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:51.525221    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95dfe3d1_42fa_41b5_a6d3_4774214d54b1.slice/crio-538883722ac1cbccca0fcccc546c2ffe23664419cc962d874aeea186fbe66e8b WatchSource:0}: Error finding container 538883722ac1cbccca0fcccc546c2ffe23664419cc962d874aeea186fbe66e8b: Status 404 returned error can't find the container with id 538883722ac1cbccca0fcccc546c2ffe23664419cc962d874aeea186fbe66e8b
Oct 30 03:57:51.831532 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.831468    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:51.832163 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.831646    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:51.846198 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.846171    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mwxht\""
Oct 30 03:57:51.846304 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.846227    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 03:57:51.846455 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.846440    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 03:57:51.846497 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.846450    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 03:57:51.846497 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:51.846451    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-vnctk\""
Oct 30 03:57:52.127640 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:52.127556    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2ldpw" event={"ID":"bee7738d-afa2-431d-ba81-2d14f142902c","Type":"ContainerStarted","Data":"822a1f9a7cf577e4192c455edd2192ac1dcb180d38395f5438a3ff7757c4e19c"}
Oct 30 03:57:52.129678 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:52.129644    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8qhbh" event={"ID":"95dfe3d1-42fa-41b5-a6d3-4774214d54b1","Type":"ContainerStarted","Data":"a992adccbd5f9c28f6e5600911a784ccefb56b83773dab1cd6719cf17e0fa815"}
Oct 30 03:57:52.129909 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:52.129711    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8qhbh" event={"ID":"95dfe3d1-42fa-41b5-a6d3-4774214d54b1","Type":"ContainerStarted","Data":"538883722ac1cbccca0fcccc546c2ffe23664419cc962d874aeea186fbe66e8b"}
Oct 30 03:57:52.132137 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:52.132110    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mctfq" event={"ID":"a6755b1f-578f-4e75-86df-be3db7533610","Type":"ContainerStarted","Data":"e73929c55e45a035187a471a0bd4973ec6348478df05c7b93f9d7b34d69b9327"}
Oct 30 03:57:53.139150 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:53.139102    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8qhbh" event={"ID":"95dfe3d1-42fa-41b5-a6d3-4774214d54b1","Type":"ContainerStarted","Data":"42d3668d80841530555d4ec65b202280e226ab44fee7c0c352dbd6c7f138ddc5"}
Oct 30 03:57:54.146471 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:54.146429    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mctfq" event={"ID":"a6755b1f-578f-4e75-86df-be3db7533610","Type":"ContainerStarted","Data":"cdf4b1b61e80ca4ecf0ce2832e7c747083d0e9c635169997950975313718c735"}
Oct 30 03:57:54.146850 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:54.146488    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mctfq" event={"ID":"a6755b1f-578f-4e75-86df-be3db7533610","Type":"ContainerStarted","Data":"55b1c80c1020409243fb069b9f36aacee29eab0841cc33d2409ab742bdbb1be9"}
Oct 30 03:57:54.146850 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:54.146610    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mctfq"
Oct 30 03:57:54.189464 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:54.189403    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-mctfq" podStartSLOduration=1.970587938 podStartE2EDuration="4.189385329s" totalImagesPullingTime="2.218797391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:54.188456837 +0000 UTC m=+33.817976602" watchObservedRunningTime="2026-10-30 03:57:54.189385329 +0000 UTC m=+33.818905072"
Oct 30 03:57:54.640442 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:54.640394    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dec34b4-fb69-459e-b130-ca1716d23765-metrics-certs\") pod \"network-metrics-daemon-dpdr4\" (UID: \"4dec34b4-fb69-459e-b130-ca1716d23765\") " pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:54.640587 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:54.640455    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bad69b26-9de8-4c2b-b19c-8a96271f5c86-original-pull-secret\") pod \"global-pull-secret-syncer-cbfm8\" (UID: \"bad69b26-9de8-4c2b-b19c-8a96271f5c86\") " pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:54.643643 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:54.643617    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bad69b26-9de8-4c2b-b19c-8a96271f5c86-original-pull-secret\") pod \"global-pull-secret-syncer-cbfm8\" (UID: \"bad69b26-9de8-4c2b-b19c-8a96271f5c86\") " pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:54.643760 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:54.643661    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dec34b4-fb69-459e-b130-ca1716d23765-metrics-certs\") pod \"network-metrics-daemon-dpdr4\" (UID: \"4dec34b4-fb69-459e-b130-ca1716d23765\") " pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:54.741105 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:54.741081    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwtgk\" (UniqueName: \"kubernetes.io/projected/d5e46634-9969-4d78-bb91-e598e950b1b0-kube-api-access-jwtgk\") pod \"network-check-target-9mktl\" (UID: \"d5e46634-9969-4d78-bb91-e598e950b1b0\") " pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:54.743366 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:54.743346    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwtgk\" (UniqueName: \"kubernetes.io/projected/d5e46634-9969-4d78-bb91-e598e950b1b0-kube-api-access-jwtgk\") pod \"network-check-target-9mktl\" (UID: \"d5e46634-9969-4d78-bb91-e598e950b1b0\") " pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:54.782123 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:54.782102    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:57:54.853053 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:54.853029    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:54.894504 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:54.894483    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:57:55.107679 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:55.106117    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cbfm8"]
Oct 30 03:57:55.152022 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:55.151960    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8qhbh" event={"ID":"95dfe3d1-42fa-41b5-a6d3-4774214d54b1","Type":"ContainerStarted","Data":"11bddbd3eaedcec43d8f27873aff0c9bb764ff441d7f562e2e5d9876d0e4de01"}
Oct 30 03:57:55.153203 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:55.153180    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cbfm8" event={"ID":"bad69b26-9de8-4c2b-b19c-8a96271f5c86","Type":"ContainerStarted","Data":"e3784334f31f6e4aa220ebcc3d15b2ce829cbf96b1454f6d591796fd249ae7c8"}
Oct 30 03:57:55.154532 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:55.154510    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2ldpw" event={"ID":"bee7738d-afa2-431d-ba81-2d14f142902c","Type":"ContainerStarted","Data":"0576e8954d352e7063db1e46721ce7b3c91a19d0fa7f9e6182bfdb5f7460a591"}
Oct 30 03:57:55.179005 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:55.178984    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9mktl"]
Oct 30 03:57:55.181566 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:55.181545    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e46634_9969_4d78_bb91_e598e950b1b0.slice/crio-68480559d0e9f8cb40e900ede9424e49bbfd02167fcc634ebfa3bdd93b29813d WatchSource:0}: Error finding container 68480559d0e9f8cb40e900ede9424e49bbfd02167fcc634ebfa3bdd93b29813d: Status 404 returned error can't find the container with id 68480559d0e9f8cb40e900ede9424e49bbfd02167fcc634ebfa3bdd93b29813d
Oct 30 03:57:55.194661 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:55.194627    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8qhbh" podStartSLOduration=2.058857222 podStartE2EDuration="5.19461612s" totalImagesPullingTime="3.135758898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:55.194470007 +0000 UTC m=+34.823989745" watchObservedRunningTime="2026-10-30 03:57:55.19461612 +0000 UTC m=+34.824135865"
Oct 30 03:57:55.227328 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:55.227310    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dpdr4"]
Oct 30 03:57:55.229996 ip-10-0-138-160 kubenswrapper[2578]: W1030 03:57:55.229972    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dec34b4_fb69_459e_b130_ca1716d23765.slice/crio-d598c99ab6258e618d4d6b0dc36296f4f132eb419b6dbb5fbc9e55d3c3d0c01f WatchSource:0}: Error finding container d598c99ab6258e618d4d6b0dc36296f4f132eb419b6dbb5fbc9e55d3c3d0c01f: Status 404 returned error can't find the container with id d598c99ab6258e618d4d6b0dc36296f4f132eb419b6dbb5fbc9e55d3c3d0c01f
Oct 30 03:57:56.160763 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:56.160718    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dpdr4" event={"ID":"4dec34b4-fb69-459e-b130-ca1716d23765","Type":"ContainerStarted","Data":"d598c99ab6258e618d4d6b0dc36296f4f132eb419b6dbb5fbc9e55d3c3d0c01f"}
Oct 30 03:57:56.162304 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:56.162173    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9mktl" event={"ID":"d5e46634-9969-4d78-bb91-e598e950b1b0","Type":"ContainerStarted","Data":"68480559d0e9f8cb40e900ede9424e49bbfd02167fcc634ebfa3bdd93b29813d"}
Oct 30 03:57:57.169785 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:57.169643    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dpdr4" event={"ID":"4dec34b4-fb69-459e-b130-ca1716d23765","Type":"ContainerStarted","Data":"09a3613d25d4717ccfb37e52ed2a3a127f8ec0d0f970c34e66d97a7569e4cc4a"}
Oct 30 03:57:57.169785 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:57.169705    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dpdr4" event={"ID":"4dec34b4-fb69-459e-b130-ca1716d23765","Type":"ContainerStarted","Data":"d5ad8c78b7c45ff7a7fbb09f60af7aa7711aa17557952f1f0bcdc8c289588e82"}
Oct 30 03:57:57.212706 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:57.212651    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dpdr4" podStartSLOduration=34.803802999 podStartE2EDuration="36.212635808s" totalImagesPullingTime="1.408832809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:57.209577795 +0000 UTC m=+36.839097539" watchObservedRunningTime="2026-10-30 03:57:57.212635808 +0000 UTC m=+36.842155552"
Oct 30 03:57:57.213739 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:57.213311    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2ldpw" podStartSLOduration=3.89271492 podStartE2EDuration="7.213303615s" totalImagesPullingTime="3.320588695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:55.23334693 +0000 UTC m=+34.862866675" watchObservedRunningTime="2026-10-30 03:57:57.213303615 +0000 UTC m=+36.842823362"
Oct 30 03:57:59.178901 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:59.178860    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9mktl" event={"ID":"d5e46634-9969-4d78-bb91-e598e950b1b0","Type":"ContainerStarted","Data":"decec18ac6b3df5f11af169a4b00139a06d38876395e6f0f706856e25037eb0f"}
Oct 30 03:57:59.179356 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:59.178994    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:57:59.220065 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:57:59.219433    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9mktl" podStartSLOduration=35.471246704 podStartE2EDuration="39.219415359s" totalImagesPullingTime="3.748168655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:59.218437157 +0000 UTC m=+38.847956903" watchObservedRunningTime="2026-10-30 03:57:59.219415359 +0000 UTC m=+38.848935108"
Oct 30 03:58:00.527251 ip-10-0-138-160 kubenswrapper[2578]: I1030 03:58:00.527157    2578 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 03:58:00.527317 ip-10-0-138-160 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 03:58:00.545578 ip-10-0-138-160 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 03:58:00.545747 ip-10-0-138-160 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 03:58:00.545916 ip-10-0-138-160 systemd[1]: kubelet.service: Consumed 2.388s CPU time.
Oct 30 03:58:00.558712 ip-10-0-138-160 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 03:58:00.644063 ip-10-0-138-160 kubenswrapper[4874]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 03:58:00.644063 ip-10-0-138-160 kubenswrapper[4874]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 03:58:00.644063 ip-10-0-138-160 kubenswrapper[4874]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 03:58:00.644063 ip-10-0-138-160 kubenswrapper[4874]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 03:58:00.646171 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646094    4874 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:58:00.646171 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646172    4874 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:58:00.646255 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646176    4874 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:58:00.646255 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646179    4874 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:58:00.646255 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646188    4874 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:58:00.646255 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646192    4874 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:58:00.646255 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646195    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:58:00.646255 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646198    4874 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:58:00.646255 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646201    4874 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:58:00.646255 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646203    4874 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:58:00.646255 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646206    4874 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:58:00.646255 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646209    4874 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:58:00.646255 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646212    4874 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:58:00.646255 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646215    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:58:00.646255 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646217    4874 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:58:00.646255 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646220    4874 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:58:00.646255 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646223    4874 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:58:00.646255 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646225    4874 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:58:00.646255 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646228    4874 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:58:00.646255 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646232    4874 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:58:00.646255 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646249    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:58:00.646255 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646252    4874 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:58:00.646736 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646255    4874 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:58:00.646736 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646258    4874 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:58:00.646736 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646261    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:58:00.646736 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646263    4874 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:58:00.646736 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646275    4874 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:58:00.646736 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646278    4874 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:58:00.646736 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646281    4874 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:58:00.646736 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646283    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:58:00.646736 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646286    4874 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:58:00.646736 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646288    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:58:00.646736 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646291    4874 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:58:00.646736 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646294    4874 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:58:00.646736 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646297    4874 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:58:00.646736 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646299    4874 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:58:00.646736 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646302    4874 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:58:00.646736 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646305    4874 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:58:00.646736 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646307    4874 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:58:00.646736 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646310    4874 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:58:00.646736 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646312    4874 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:58:00.647185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646315    4874 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:58:00.647185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646318    4874 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:58:00.647185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646320    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:58:00.647185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646323    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:58:00.647185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646325    4874 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:58:00.647185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646328    4874 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:58:00.647185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646330    4874 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:58:00.647185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646333    4874 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:58:00.647185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646335    4874 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:58:00.647185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646338    4874 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:58:00.647185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646341    4874 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:58:00.647185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646344    4874 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:58:00.647185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646346    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:58:00.647185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646349    4874 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:58:00.647185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646352    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:58:00.647185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646354    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:58:00.647185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646357    4874 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:58:00.647185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646360    4874 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:58:00.647185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646362    4874 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:58:00.647185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646365    4874 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:58:00.647670 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646368    4874 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:58:00.647670 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646370    4874 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:58:00.647670 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646373    4874 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:58:00.647670 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646375    4874 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:58:00.647670 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646378    4874 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:58:00.647670 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646381    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:58:00.647670 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646384    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:58:00.647670 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646387    4874 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:58:00.647670 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646389    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:58:00.647670 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646392    4874 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:58:00.647670 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646394    4874 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:58:00.647670 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646397    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:58:00.647670 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646400    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:58:00.647670 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646402    4874 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:58:00.647670 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646405    4874 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:58:00.647670 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646407    4874 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:58:00.647670 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646410    4874 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:58:00.647670 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646413    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:58:00.647670 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646415    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:58:00.648110 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646418    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:58:00.648110 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646420    4874 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:58:00.648110 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646423    4874 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:58:00.648110 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646425    4874 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:58:00.648110 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646428    4874 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:58:00.648110 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646430    4874 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:58:00.648110 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646433    4874 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:58:00.648110 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646436    4874 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:58:00.648110 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646439    4874 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:58:00.648110 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646442    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:58:00.648110 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646444    4874 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:58:00.648110 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646447    4874 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:58:00.648110 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646449    4874 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:58:00.648110 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646452    4874 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:58:00.648110 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646454    4874 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:58:00.648110 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646457    4874 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:58:00.648110 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646460    4874 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:58:00.648110 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646462    4874 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:58:00.648110 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646465    4874 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:58:00.648110 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646468    4874 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:58:00.648606 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646470    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:58:00.648606 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646475    4874 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:58:00.648606 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646479    4874 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:58:00.648606 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646481    4874 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:58:00.648606 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646484    4874 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:58:00.648606 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.646487    4874 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:58:00.648606 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647250    4874 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:58:00.648606 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647258    4874 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:58:00.648606 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647262    4874 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:58:00.648606 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647266    4874 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:58:00.648606 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647269    4874 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:58:00.648606 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647272    4874 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:58:00.648606 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647274    4874 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:58:00.648606 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647277    4874 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:58:00.648606 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647279    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:58:00.648606 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647282    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:58:00.648606 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647284    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:58:00.648606 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647287    4874 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:58:00.648606 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647290    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:58:00.649281 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647293    4874 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:58:00.649281 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647296    4874 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:58:00.649281 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647301    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:58:00.649281 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647304    4874 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:58:00.649281 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647307    4874 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:58:00.649281 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647310    4874 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:58:00.649281 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647313    4874 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:58:00.649281 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647315    4874 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:58:00.649281 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647318    4874 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:58:00.649281 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647321    4874 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:58:00.649281 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647323    4874 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:58:00.649281 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647326    4874 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:58:00.649281 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647329    4874 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:58:00.649281 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647331    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:58:00.649281 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647334    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:58:00.649281 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647336    4874 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:58:00.649281 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647339    4874 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:58:00.649281 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647341    4874 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:58:00.649281 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647355    4874 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:58:00.649281 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647360    4874 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:58:00.649761 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647363    4874 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:58:00.649761 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647365    4874 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:58:00.649761 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647368    4874 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:58:00.649761 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647372    4874 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:58:00.649761 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647374    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:58:00.649761 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647377    4874 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:58:00.649761 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647379    4874 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:58:00.649761 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647382    4874 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:58:00.649761 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647385    4874 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:58:00.649761 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647388    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:58:00.649761 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647390    4874 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:58:00.649761 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647393    4874 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:58:00.649761 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647396    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:58:00.649761 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647398    4874 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:58:00.649761 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647402    4874 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:58:00.649761 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647405    4874 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:58:00.649761 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647407    4874 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:58:00.649761 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647410    4874 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:58:00.649761 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647412    4874 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:58:00.649761 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647415    4874 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:58:00.650272 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647418    4874 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:58:00.650272 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647420    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:58:00.650272 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647423    4874 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:58:00.650272 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647425    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:58:00.650272 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647428    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:58:00.650272 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647431    4874 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:58:00.650272 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647434    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:58:00.650272 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647437    4874 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:58:00.650272 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647441    4874 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:58:00.650272 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647445    4874 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:58:00.650272 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647449    4874 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:58:00.650272 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647452    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:58:00.650272 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647456    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:58:00.650272 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647459    4874 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:58:00.650272 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647462    4874 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:58:00.650272 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647464    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:58:00.650272 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647467    4874 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:58:00.650272 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647471    4874 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:58:00.650697 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647475    4874 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:58:00.650697 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647478    4874 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:58:00.650697 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647480    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:58:00.650697 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647483    4874 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:58:00.650697 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647485    4874 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:58:00.650697 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647488    4874 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:58:00.650697 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647491    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:58:00.650697 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647493    4874 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:58:00.650697 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647498    4874 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:58:00.650697 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647500    4874 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:58:00.650697 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647503    4874 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:58:00.650697 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647506    4874 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:58:00.650697 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647509    4874 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:58:00.650697 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647511    4874 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:58:00.650697 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647514    4874 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:58:00.650697 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647517    4874 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:58:00.650697 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647520    4874 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:58:00.650697 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647522    4874 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:58:00.650697 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647525    4874 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:58:00.650697 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647527    4874 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:58:00.651204 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647530    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:58:00.651204 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647533    4874 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:58:00.651204 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647535    4874 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:58:00.651204 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647537    4874 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:58:00.651204 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647540    4874 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:58:00.651204 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647542    4874 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:58:00.651204 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647546    4874 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:58:00.651204 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647548    4874 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:58:00.651204 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647551    4874 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:58:00.651204 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647553    4874 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:58:00.651204 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647556    4874 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:58:00.651204 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647560    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:58:00.651204 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647563    4874 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:58:00.651204 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647565    4874 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:58:00.651204 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.647567    4874 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:58:00.651204 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648048    4874 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:58:00.651204 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648053    4874 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:58:00.651204 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648056    4874 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:58:00.651204 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648059    4874 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:58:00.651204 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648061    4874 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:58:00.651792 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648066    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:58:00.651792 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648068    4874 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:58:00.651792 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648071    4874 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:58:00.651792 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648074    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:58:00.651792 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648076    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:58:00.651792 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648085    4874 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:58:00.651792 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648088    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:58:00.651792 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648096    4874 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:58:00.651792 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648098    4874 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:58:00.651792 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648101    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:58:00.651792 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648104    4874 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:58:00.651792 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648106    4874 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:58:00.651792 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648109    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:58:00.651792 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648111    4874 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:58:00.651792 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648114    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:58:00.651792 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648117    4874 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:58:00.651792 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648120    4874 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:58:00.651792 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648122    4874 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:58:00.651792 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648126    4874 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:58:00.651792 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648129    4874 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:58:00.652562 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648132    4874 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:58:00.652562 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648134    4874 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:58:00.652562 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648136    4874 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:58:00.652562 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648145    4874 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:58:00.652562 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648148    4874 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:58:00.652562 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648150    4874 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:58:00.652562 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648153    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:58:00.652562 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648155    4874 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:58:00.652562 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648158    4874 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:58:00.652562 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648161    4874 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:58:00.652562 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648163    4874 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:58:00.652562 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648166    4874 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:58:00.652562 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648169    4874 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:58:00.652562 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648173    4874 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:58:00.652562 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648176    4874 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:58:00.652562 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648179    4874 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:58:00.652562 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648182    4874 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:58:00.652562 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648184    4874 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:58:00.652562 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648186    4874 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:58:00.652562 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648189    4874 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:58:00.653418 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648192    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:58:00.653418 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648194    4874 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:58:00.653418 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648197    4874 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:58:00.653418 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648199    4874 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:58:00.653418 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648202    4874 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:58:00.653418 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648204    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:58:00.653418 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648207    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:58:00.653418 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648209    4874 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:58:00.653418 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648212    4874 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:58:00.653418 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648214    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:58:00.653418 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648218    4874 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:58:00.653418 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648222    4874 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:58:00.653418 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648226    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:58:00.653418 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648229    4874 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:58:00.653418 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648232    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:58:00.653418 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648234    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:58:00.653418 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648256    4874 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:58:00.653418 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648259    4874 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:58:00.653418 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648261    4874 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:58:00.654349 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648264    4874 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:58:00.654349 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648266    4874 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:58:00.654349 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648269    4874 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:58:00.654349 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648271    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:58:00.654349 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648274    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:58:00.654349 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648277    4874 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:58:00.654349 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648279    4874 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:58:00.654349 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648282    4874 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:58:00.654349 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648285    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:58:00.654349 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648288    4874 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:58:00.654349 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648291    4874 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:58:00.654349 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648293    4874 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:58:00.654349 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648296    4874 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:58:00.654349 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648298    4874 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:58:00.654349 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648301    4874 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:58:00.654349 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648303    4874 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:58:00.654349 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648306    4874 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:58:00.654349 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648308    4874 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:58:00.654349 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648311    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:58:00.655182 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648314    4874 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:58:00.655182 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648316    4874 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:58:00.655182 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648319    4874 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:58:00.655182 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648321    4874 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:58:00.655182 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648325    4874 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:58:00.655182 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648328    4874 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:58:00.655182 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648331    4874 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:58:00.655182 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648334    4874 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:58:00.655182 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648336    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:58:00.655182 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648339    4874 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:58:00.655182 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648341    4874 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:58:00.655182 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648350    4874 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:58:00.655182 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648353    4874 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:58:00.655182 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648356    4874 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:58:00.655182 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648358    4874 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:58:00.655182 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648361    4874 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:58:00.655182 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648363    4874 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:58:00.655182 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648366    4874 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:58:00.655182 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648369    4874 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:58:00.655182 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648371    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:58:00.656092 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648374    4874 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:58:00.656092 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648376    4874 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:58:00.656092 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.648379    4874 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:58:00.656092 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648467    4874 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 03:58:00.656092 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648474    4874 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 03:58:00.656092 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648486    4874 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 03:58:00.656092 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648491    4874 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 03:58:00.656092 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648495    4874 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 03:58:00.656092 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648499    4874 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 03:58:00.656092 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648503    4874 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 03:58:00.656092 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648507    4874 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 03:58:00.656092 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648510    4874 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 03:58:00.656092 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648513    4874 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 03:58:00.656092 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648517    4874 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 03:58:00.656092 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648520    4874 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 03:58:00.656092 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648523    4874 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 03:58:00.656092 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648527    4874 flags.go:64] FLAG: --cgroup-root=""
Oct 30 03:58:00.656092 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648531    4874 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 03:58:00.656092 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648534    4874 flags.go:64] FLAG: --client-ca-file=""
Oct 30 03:58:00.656092 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648537    4874 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 03:58:00.656092 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648542    4874 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 03:58:00.656092 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648548    4874 flags.go:64] FLAG: --cluster-domain=""
Oct 30 03:58:00.656092 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648551    4874 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 03:58:00.656092 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648554    4874 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 03:58:00.657157 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648564    4874 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 03:58:00.657157 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648568    4874 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 03:58:00.657157 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648572    4874 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 03:58:00.657157 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648575    4874 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 03:58:00.657157 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648579    4874 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 03:58:00.657157 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648582    4874 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 03:58:00.657157 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648585    4874 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 03:58:00.657157 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648588    4874 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 03:58:00.657157 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648591    4874 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 03:58:00.657157 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648594    4874 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 03:58:00.657157 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648597    4874 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 03:58:00.657157 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648601    4874 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 03:58:00.657157 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648604    4874 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 03:58:00.657157 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648607    4874 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 03:58:00.657157 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648611    4874 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 03:58:00.657157 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648614    4874 flags.go:64] FLAG: --enable-server="true"
Oct 30 03:58:00.657157 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648617    4874 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 03:58:00.657157 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648624    4874 flags.go:64] FLAG: --event-burst="100"
Oct 30 03:58:00.657157 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648627    4874 flags.go:64] FLAG: --event-qps="50"
Oct 30 03:58:00.657157 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648630    4874 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 03:58:00.657157 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648633    4874 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 03:58:00.657157 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648636    4874 flags.go:64] FLAG: --eviction-hard=""
Oct 30 03:58:00.657157 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648641    4874 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 03:58:00.657157 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648644    4874 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 03:58:00.658249 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648648    4874 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 03:58:00.658249 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648651    4874 flags.go:64] FLAG: --eviction-soft=""
Oct 30 03:58:00.658249 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648654    4874 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 03:58:00.658249 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648657    4874 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 03:58:00.658249 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648661    4874 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 03:58:00.658249 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648664    4874 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 03:58:00.658249 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648667    4874 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 03:58:00.658249 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648670    4874 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 03:58:00.658249 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648673    4874 flags.go:64] FLAG: --feature-gates=""
Oct 30 03:58:00.658249 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648682    4874 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 03:58:00.658249 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648686    4874 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 03:58:00.658249 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648689    4874 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 03:58:00.658249 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648692    4874 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 03:58:00.658249 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648695    4874 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 03:58:00.658249 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648698    4874 flags.go:64] FLAG: --help="false"
Oct 30 03:58:00.658249 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648701    4874 flags.go:64] FLAG: --hostname-override="ip-10-0-138-160.ec2.internal"
Oct 30 03:58:00.658249 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648704    4874 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 03:58:00.658249 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648707    4874 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 03:58:00.658249 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648709    4874 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 03:58:00.658249 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648713    4874 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 03:58:00.658249 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648716    4874 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 03:58:00.658249 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648719    4874 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 03:58:00.658249 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648722    4874 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 03:58:00.659170 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648725    4874 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 03:58:00.659170 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648728    4874 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 03:58:00.659170 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648731    4874 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 03:58:00.659170 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648734    4874 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 03:58:00.659170 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648737    4874 flags.go:64] FLAG: --kube-reserved=""
Oct 30 03:58:00.659170 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648740    4874 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 03:58:00.659170 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648743    4874 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 03:58:00.659170 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648746    4874 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 03:58:00.659170 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648748    4874 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 03:58:00.659170 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648752    4874 flags.go:64] FLAG: --lock-file=""
Oct 30 03:58:00.659170 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648755    4874 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 03:58:00.659170 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648758    4874 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 03:58:00.659170 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648761    4874 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 03:58:00.659170 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648767    4874 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 03:58:00.659170 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648771    4874 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 03:58:00.659170 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648774    4874 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 03:58:00.659170 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648777    4874 flags.go:64] FLAG: --logging-format="text"
Oct 30 03:58:00.659170 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648780    4874 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 03:58:00.659170 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648783    4874 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 03:58:00.659170 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648796    4874 flags.go:64] FLAG: --manifest-url=""
Oct 30 03:58:00.659170 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648799    4874 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 03:58:00.659170 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648803    4874 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 03:58:00.659170 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648807    4874 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 03:58:00.659170 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648810    4874 flags.go:64] FLAG: --max-pods="110"
Oct 30 03:58:00.659170 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648813    4874 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648816    4874 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648820    4874 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648823    4874 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648825    4874 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648828    4874 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648831    4874 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648838    4874 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648841    4874 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648844    4874 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648847    4874 flags.go:64] FLAG: --pod-cidr=""
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648850    4874 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648853    4874 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648856    4874 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648859    4874 flags.go:64] FLAG: --port="10250"
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648862    4874 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648865    4874 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05d99f2bdb0bdb8d8"
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648868    4874 flags.go:64] FLAG: --qos-reserved=""
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648871    4874 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648874    4874 flags.go:64] FLAG: --register-node="true"
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648877    4874 flags.go:64] FLAG: --register-with-taints=""
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648882    4874 flags.go:64] FLAG: --registry-burst="10"
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648885    4874 flags.go:64] FLAG: --registry-qps="5"
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648888    4874 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648891    4874 flags.go:64] FLAG: --reserved-memory=""
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648894    4874 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 03:58:00.660159 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648898    4874 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 03:58:00.660840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648901    4874 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 03:58:00.660840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648910    4874 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 03:58:00.660840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648913    4874 flags.go:64] FLAG: --runonce="false"
Oct 30 03:58:00.660840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648916    4874 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 03:58:00.660840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648919    4874 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 03:58:00.660840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648922    4874 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 03:58:00.660840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648924    4874 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 03:58:00.660840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648927    4874 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 03:58:00.660840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648931    4874 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 03:58:00.660840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648934    4874 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 03:58:00.660840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648937    4874 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 03:58:00.660840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648939    4874 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 03:58:00.660840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648942    4874 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 03:58:00.660840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648945    4874 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 03:58:00.660840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648948    4874 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 03:58:00.660840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648951    4874 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 03:58:00.660840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648954    4874 flags.go:64] FLAG: --system-cgroups=""
Oct 30 03:58:00.660840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648957    4874 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 03:58:00.660840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648962    4874 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 03:58:00.660840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648965    4874 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 03:58:00.660840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648967    4874 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 03:58:00.660840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648974    4874 flags.go:64] FLAG: --tls-min-version=""
Oct 30 03:58:00.660840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648977    4874 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 03:58:00.660840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648979    4874 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 03:58:00.660840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648983    4874 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 03:58:00.661451 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648985    4874 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 03:58:00.661451 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648991    4874 flags.go:64] FLAG: --v="2"
Oct 30 03:58:00.661451 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648995    4874 flags.go:64] FLAG: --version="false"
Oct 30 03:58:00.661451 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.648999    4874 flags.go:64] FLAG: --vmodule=""
Oct 30 03:58:00.661451 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.649003    4874 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 03:58:00.661451 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.649006    4874 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 03:58:00.661451 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649186    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:58:00.661451 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649190    4874 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:58:00.661451 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649193    4874 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:58:00.661451 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649201    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:58:00.661451 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649204    4874 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:58:00.661451 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649207    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:58:00.661451 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649211    4874 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:58:00.661451 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649215    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:58:00.661451 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649218    4874 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:58:00.661451 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649222    4874 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:58:00.661451 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649226    4874 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:58:00.661451 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649230    4874 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:58:00.661451 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649233    4874 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:58:00.661451 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649249    4874 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:58:00.661922 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649252    4874 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:58:00.661922 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649255    4874 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:58:00.661922 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649257    4874 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:58:00.661922 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649260    4874 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:58:00.661922 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649262    4874 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:58:00.661922 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649265    4874 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:58:00.661922 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649268    4874 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:58:00.661922 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649270    4874 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:58:00.661922 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649273    4874 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:58:00.661922 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649277    4874 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:58:00.661922 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649280    4874 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:58:00.661922 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649283    4874 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:58:00.661922 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649286    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:58:00.661922 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649290    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:58:00.661922 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649293    4874 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:58:00.661922 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649295    4874 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:58:00.661922 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649298    4874 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:58:00.661922 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649300    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:58:00.661922 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649303    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:58:00.662530 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649306    4874 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:58:00.662530 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649309    4874 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:58:00.662530 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649311    4874 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:58:00.662530 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649314    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:58:00.662530 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649317    4874 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:58:00.662530 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649320    4874 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:58:00.662530 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649322    4874 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:58:00.662530 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649325    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:58:00.662530 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649327    4874 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:58:00.662530 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649330    4874 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:58:00.662530 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649333    4874 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:58:00.662530 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649335    4874 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:58:00.662530 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649338    4874 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:58:00.662530 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649340    4874 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:58:00.662530 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649343    4874 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:58:00.662530 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649346    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:58:00.662530 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649348    4874 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:58:00.662530 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649351    4874 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:58:00.662530 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649353    4874 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:58:00.662530 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649356    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:58:00.663121 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649358    4874 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:58:00.663121 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649361    4874 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:58:00.663121 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649366    4874 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:58:00.663121 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649369    4874 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:58:00.663121 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649371    4874 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:58:00.663121 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649374    4874 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:58:00.663121 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649378    4874 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:58:00.663121 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649381    4874 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:58:00.663121 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649384    4874 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:58:00.663121 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649386    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:58:00.663121 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649389    4874 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:58:00.663121 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649392    4874 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:58:00.663121 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649394    4874 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:58:00.663121 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649397    4874 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:58:00.663121 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649399    4874 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:58:00.663121 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649402    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:58:00.663121 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649405    4874 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:58:00.663121 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649407    4874 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:58:00.663121 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649410    4874 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:58:00.663121 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649413    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:58:00.663779 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649416    4874 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:58:00.663779 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649418    4874 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:58:00.663779 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649421    4874 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:58:00.663779 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649423    4874 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:58:00.663779 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649426    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:58:00.663779 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649429    4874 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:58:00.663779 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649431    4874 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:58:00.663779 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649434    4874 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:58:00.663779 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649437    4874 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:58:00.663779 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649439    4874 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:58:00.663779 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649442    4874 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:58:00.663779 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649444    4874 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:58:00.663779 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649447    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:58:00.663779 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649450    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:58:00.663779 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649454    4874 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:58:00.663779 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649456    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:58:00.663779 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649459    4874 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:58:00.663779 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649462    4874 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:58:00.663779 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649466    4874 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:58:00.663779 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649469    4874 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:58:00.664289 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649472    4874 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:58:00.664289 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649474    4874 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:58:00.664289 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649477    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:58:00.664289 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649480    4874 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:58:00.664289 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649482    4874 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:58:00.664289 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649485    4874 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:58:00.664289 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649487    4874 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:58:00.664289 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649490    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:58:00.664289 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649493    4874 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:58:00.664289 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649495    4874 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:58:00.664289 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649498    4874 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:58:00.664289 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649501    4874 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:58:00.664289 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649503    4874 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:58:00.664289 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.649529    4874 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 03:58:00.664289 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649565    4874 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:58:00.664289 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649569    4874 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:58:00.664673 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649572    4874 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:58:00.664673 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649575    4874 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:58:00.664673 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649578    4874 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:58:00.664673 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649580    4874 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:58:00.664673 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649583    4874 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:58:00.664673 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649586    4874 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:58:00.664673 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649589    4874 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:58:00.664673 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649591    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:58:00.664673 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649594    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:58:00.664673 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649596    4874 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:58:00.664673 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649599    4874 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:58:00.664673 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649603    4874 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:58:00.664673 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649606    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:58:00.664673 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649608    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:58:00.664673 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649612    4874 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:58:00.664673 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649615    4874 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:58:00.664673 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649617    4874 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:58:00.664673 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649620    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:58:00.664673 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649622    4874 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:58:00.665175 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649625    4874 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:58:00.665175 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649627    4874 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:58:00.665175 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649630    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:58:00.665175 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649633    4874 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:58:00.665175 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649635    4874 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:58:00.665175 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649638    4874 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:58:00.665175 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649640    4874 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:58:00.665175 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649643    4874 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:58:00.665175 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649646    4874 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:58:00.665175 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649649    4874 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:58:00.665175 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649651    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:58:00.665175 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649654    4874 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:58:00.665175 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649656    4874 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:58:00.665175 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649659    4874 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:58:00.665175 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649661    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:58:00.665175 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649664    4874 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:58:00.665175 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649666    4874 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:58:00.665175 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649669    4874 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:58:00.665175 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649671    4874 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:58:00.665175 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649674    4874 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:58:00.665724 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649677    4874 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:58:00.665724 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649680    4874 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:58:00.665724 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649682    4874 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:58:00.665724 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649685    4874 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:58:00.665724 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649689    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:58:00.665724 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649691    4874 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:58:00.665724 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649694    4874 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:58:00.665724 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649698    4874 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:58:00.665724 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649700    4874 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:58:00.665724 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649703    4874 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:58:00.665724 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649705    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:58:00.665724 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649708    4874 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:58:00.665724 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649710    4874 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:58:00.665724 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649713    4874 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:58:00.665724 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649716    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:58:00.665724 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649719    4874 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:58:00.665724 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649722    4874 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:58:00.665724 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649724    4874 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:58:00.665724 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649727    4874 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:58:00.665724 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649729    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:58:00.666225 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649732    4874 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:58:00.666225 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649735    4874 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:58:00.666225 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649738    4874 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:58:00.666225 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649740    4874 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:58:00.666225 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649743    4874 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:58:00.666225 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649746    4874 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:58:00.666225 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649749    4874 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:58:00.666225 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649751    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:58:00.666225 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649754    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:58:00.666225 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649756    4874 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:58:00.666225 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649759    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:58:00.666225 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649761    4874 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:58:00.666225 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649764    4874 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:58:00.666225 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649767    4874 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:58:00.666225 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649770    4874 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:58:00.666225 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649772    4874 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:58:00.666225 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649776    4874 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:58:00.666225 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649780    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:58:00.666225 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649783    4874 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:58:00.666225 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649787    4874 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:58:00.666728 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649789    4874 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:58:00.666728 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649792    4874 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:58:00.666728 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649795    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:58:00.666728 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649797    4874 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:58:00.666728 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649800    4874 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:58:00.666728 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649802    4874 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:58:00.666728 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649805    4874 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:58:00.666728 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649808    4874 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:58:00.666728 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649810    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:58:00.666728 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649813    4874 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:58:00.666728 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649815    4874 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:58:00.666728 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649818    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:58:00.666728 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649820    4874 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:58:00.666728 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649824    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:58:00.666728 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649827    4874 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:58:00.666728 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649831    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:58:00.666728 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649834    4874 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:58:00.666728 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649838    4874 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:58:00.666728 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649842    4874 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:58:00.667183 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649846    4874 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:58:00.667183 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649850    4874 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:58:00.667183 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649852    4874 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:58:00.667183 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649855    4874 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:58:00.667183 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649858    4874 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:58:00.667183 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.649860    4874 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:58:00.667183 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.654336    4874 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 03:58:00.667183 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.654356    4874 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 03:58:00.667183 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.654380    4874 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 03:58:00.667183 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.654388    4874 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 03:58:00.667183 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654472    4874 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:58:00.667183 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654480    4874 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:58:00.667183 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654485    4874 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:58:00.667183 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654490    4874 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:58:00.667183 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654496    4874 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:58:00.667183 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654501    4874 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:58:00.667183 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654505    4874 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:58:00.667183 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654510    4874 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:58:00.667183 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654515    4874 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:58:00.667183 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654519    4874 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:58:00.667703 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654524    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:58:00.667703 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654528    4874 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:58:00.667703 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654534    4874 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:58:00.667703 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654539    4874 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:58:00.667703 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654543    4874 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:58:00.667703 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654548    4874 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:58:00.667703 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654553    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:58:00.667703 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654556    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:58:00.667703 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654561    4874 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:58:00.667703 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654566    4874 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:58:00.667703 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654570    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:58:00.667703 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654575    4874 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:58:00.667703 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654579    4874 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:58:00.667703 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654583    4874 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:58:00.667703 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654587    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:58:00.667703 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654592    4874 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:58:00.667703 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654596    4874 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:58:00.667703 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654600    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:58:00.667703 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654605    4874 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:58:00.668185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654610    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:58:00.668185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654614    4874 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:58:00.668185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654618    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:58:00.668185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654623    4874 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:58:00.668185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654627    4874 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:58:00.668185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654631    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:58:00.668185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654635    4874 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:58:00.668185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654640    4874 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:58:00.668185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654644    4874 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:58:00.668185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654649    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:58:00.668185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654653    4874 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:58:00.668185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654657    4874 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:58:00.668185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654662    4874 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:58:00.668185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654666    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:58:00.668185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654670    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:58:00.668185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654674    4874 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:58:00.668185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654678    4874 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:58:00.668185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654684    4874 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:58:00.668185 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654689    4874 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:58:00.668657 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654693    4874 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:58:00.668657 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654697    4874 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:58:00.668657 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654702    4874 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:58:00.668657 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654706    4874 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:58:00.668657 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654710    4874 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:58:00.668657 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654714    4874 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:58:00.668657 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654721    4874 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:58:00.668657 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654727    4874 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:58:00.668657 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654732    4874 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:58:00.668657 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654737    4874 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:58:00.668657 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654741    4874 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:58:00.668657 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654745    4874 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:58:00.668657 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654749    4874 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:58:00.668657 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654754    4874 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:58:00.668657 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654758    4874 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:58:00.668657 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654763    4874 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:58:00.668657 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654766    4874 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:58:00.668657 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654771    4874 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:58:00.668657 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654775    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:58:00.669106 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654780    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:58:00.669106 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654784    4874 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:58:00.669106 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654789    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:58:00.669106 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654793    4874 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:58:00.669106 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654798    4874 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:58:00.669106 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654802    4874 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:58:00.669106 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654806    4874 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:58:00.669106 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654810    4874 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:58:00.669106 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654815    4874 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:58:00.669106 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654819    4874 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:58:00.669106 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654826    4874 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:58:00.669106 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654832    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:58:00.669106 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654838    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:58:00.669106 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654843    4874 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:58:00.669106 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654847    4874 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:58:00.669106 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654851    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:58:00.669106 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654855    4874 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:58:00.669106 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654860    4874 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:58:00.669106 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654864    4874 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:58:00.669691 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654869    4874 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:58:00.669691 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654874    4874 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:58:00.669691 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654878    4874 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:58:00.669691 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654882    4874 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:58:00.669691 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654887    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:58:00.669691 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654891    4874 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:58:00.669691 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654895    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:58:00.669691 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654899    4874 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:58:00.669691 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654903    4874 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:58:00.669691 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654908    4874 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:58:00.669691 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654913    4874 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:58:00.669691 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654917    4874 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:58:00.669691 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654921    4874 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:58:00.669691 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654926    4874 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:58:00.669691 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654930    4874 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:58:00.669691 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654934    4874 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:58:00.669691 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654938    4874 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:58:00.669691 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654942    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:58:00.669691 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654946    4874 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:58:00.669691 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.654951    4874 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:58:00.670311 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.654990    4874 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 03:58:00.670311 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655222    4874 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:58:00.670311 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655232    4874 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:58:00.670311 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655254    4874 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:58:00.670311 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655260    4874 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:58:00.670311 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655265    4874 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:58:00.670311 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655269    4874 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:58:00.670311 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655275    4874 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:58:00.670311 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655279    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:58:00.670311 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655282    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:58:00.670311 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655287    4874 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:58:00.670311 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655291    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:58:00.670311 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655295    4874 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:58:00.670311 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655300    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:58:00.670311 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655304    4874 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:58:00.670311 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655308    4874 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:58:00.670810 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655313    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:58:00.670810 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655317    4874 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:58:00.670810 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655321    4874 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:58:00.670810 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655325    4874 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:58:00.670810 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655329    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:58:00.670810 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655334    4874 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:58:00.670810 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655340    4874 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:58:00.670810 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655345    4874 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:58:00.670810 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655349    4874 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:58:00.670810 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655353    4874 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:58:00.670810 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655357    4874 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:58:00.670810 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655361    4874 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:58:00.670810 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655365    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:58:00.670810 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655369    4874 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:58:00.670810 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655374    4874 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:58:00.670810 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655378    4874 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:58:00.670810 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655382    4874 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:58:00.670810 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655386    4874 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:58:00.670810 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655391    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:58:00.670810 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655395    4874 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:58:00.671582 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655400    4874 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:58:00.671582 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655404    4874 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:58:00.671582 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655408    4874 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:58:00.671582 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655412    4874 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:58:00.671582 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655417    4874 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:58:00.671582 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655421    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:58:00.671582 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655425    4874 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:58:00.671582 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655430    4874 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:58:00.671582 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655434    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:58:00.671582 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655438    4874 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:58:00.671582 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655442    4874 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:58:00.671582 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655451    4874 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:58:00.671582 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655455    4874 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:58:00.671582 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655460    4874 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:58:00.671582 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655467    4874 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:58:00.671582 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655473    4874 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:58:00.671582 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655478    4874 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:58:00.671582 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655483    4874 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:58:00.671582 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655488    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:58:00.672194 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655493    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:58:00.672194 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655497    4874 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:58:00.672194 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655502    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:58:00.672194 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655506    4874 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:58:00.672194 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655511    4874 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:58:00.672194 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655515    4874 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:58:00.672194 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655519    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:58:00.672194 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655524    4874 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:58:00.672194 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655528    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:58:00.672194 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655532    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:58:00.672194 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655536    4874 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:58:00.672194 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655541    4874 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:58:00.672194 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655545    4874 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:58:00.672194 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655549    4874 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:58:00.672194 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655554    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:58:00.672194 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655558    4874 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:58:00.672194 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655562    4874 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:58:00.672194 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655567    4874 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:58:00.672194 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655571    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:58:00.672721 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655575    4874 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:58:00.672721 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655579    4874 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:58:00.672721 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655584    4874 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:58:00.672721 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655589    4874 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:58:00.672721 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655593    4874 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:58:00.672721 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655599    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:58:00.672721 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655603    4874 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:58:00.672721 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655607    4874 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:58:00.672721 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655612    4874 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:58:00.672721 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655616    4874 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:58:00.672721 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655620    4874 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:58:00.672721 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655625    4874 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:58:00.672721 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655629    4874 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:58:00.672721 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655634    4874 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:58:00.672721 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655639    4874 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:58:00.672721 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655645    4874 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:58:00.672721 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655652    4874 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:58:00.672721 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655657    4874 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:58:00.672721 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655661    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:58:00.673180 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655665    4874 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:58:00.673180 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655669    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:58:00.673180 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655674    4874 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:58:00.673180 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655678    4874 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:58:00.673180 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655683    4874 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:58:00.673180 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655687    4874 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:58:00.673180 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655691    4874 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:58:00.673180 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655697    4874 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:58:00.673180 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655702    4874 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:58:00.673180 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655706    4874 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:58:00.673180 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655710    4874 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:58:00.673180 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655714    4874 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:58:00.673180 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655719    4874 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:58:00.673180 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655723    4874 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:58:00.673180 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.655763    4874 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 03:58:00.673180 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655813    4874 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:58:00.673180 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655820    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:58:00.673633 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655826    4874 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:58:00.673633 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655831    4874 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:58:00.673633 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655838    4874 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:58:00.673633 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655844    4874 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:58:00.673633 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655848    4874 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:58:00.673633 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655853    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:58:00.673633 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655858    4874 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:58:00.673633 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655862    4874 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:58:00.673633 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655867    4874 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:58:00.673633 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655871    4874 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:58:00.673633 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655875    4874 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:58:00.673633 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655880    4874 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:58:00.673633 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655885    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:58:00.673633 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655890    4874 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:58:00.673633 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655894    4874 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:58:00.673633 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655898    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:58:00.673633 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655902    4874 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:58:00.673633 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655907    4874 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:58:00.673633 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655911    4874 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:58:00.673633 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655915    4874 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:58:00.674135 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655920    4874 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:58:00.674135 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655928    4874 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:58:00.674135 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655935    4874 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:58:00.674135 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655941    4874 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:58:00.674135 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655946    4874 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:58:00.674135 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655950    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:58:00.674135 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655956    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:58:00.674135 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655960    4874 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:58:00.674135 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655965    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:58:00.674135 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655970    4874 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:58:00.674135 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655974    4874 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:58:00.674135 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655979    4874 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:58:00.674135 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655983    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:58:00.674135 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655987    4874 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:58:00.674135 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655993    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:58:00.674135 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.655997    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:58:00.674135 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656001    4874 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:58:00.674135 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656006    4874 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:58:00.674135 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656010    4874 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:58:00.674679 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656015    4874 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:58:00.674679 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656019    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:58:00.674679 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656023    4874 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:58:00.674679 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656028    4874 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:58:00.674679 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656032    4874 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:58:00.674679 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656037    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:58:00.674679 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656041    4874 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:58:00.674679 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656045    4874 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:58:00.674679 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656050    4874 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:58:00.674679 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656055    4874 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:58:00.674679 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656059    4874 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:58:00.674679 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656064    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:58:00.674679 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656068    4874 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:58:00.674679 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656072    4874 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:58:00.674679 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656077    4874 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:58:00.674679 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656082    4874 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:58:00.674679 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656087    4874 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:58:00.674679 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656090    4874 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:58:00.674679 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656094    4874 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:58:00.675426 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656099    4874 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:58:00.675426 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656103    4874 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:58:00.675426 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656109    4874 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:58:00.675426 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656115    4874 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:58:00.675426 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656119    4874 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:58:00.675426 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656123    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:58:00.675426 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656128    4874 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:58:00.675426 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656132    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:58:00.675426 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656138    4874 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:58:00.675426 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656143    4874 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:58:00.675426 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656147    4874 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:58:00.675426 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656151    4874 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:58:00.675426 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656156    4874 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:58:00.675426 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656161    4874 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:58:00.675426 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656165    4874 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:58:00.675426 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656169    4874 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:58:00.675426 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656173    4874 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:58:00.675426 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656177    4874 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:58:00.675426 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656181    4874 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:58:00.675426 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656187    4874 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:58:00.676201 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656191    4874 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:58:00.676201 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656196    4874 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:58:00.676201 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656200    4874 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:58:00.676201 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656204    4874 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:58:00.676201 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656208    4874 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:58:00.676201 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656212    4874 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:58:00.676201 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656217    4874 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:58:00.676201 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656222    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:58:00.676201 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656226    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:58:00.676201 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656231    4874 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:58:00.676201 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656235    4874 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:58:00.676201 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656256    4874 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:58:00.676201 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656260    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:58:00.676201 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656264    4874 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:58:00.676201 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656268    4874 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:58:00.676201 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656272    4874 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:58:00.676201 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656277    4874 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:58:00.676201 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656281    4874 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:58:00.676201 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656286    4874 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:58:00.676201 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656290    4874 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:58:00.676848 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656296    4874 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:58:00.676848 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656300    4874 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:58:00.676848 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656306    4874 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:58:00.676848 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656311    4874 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:58:00.676848 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656315    4874 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:58:00.676848 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:00.656319    4874 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:58:00.676848 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.656417    4874 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 03:58:00.676848 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.657904    4874 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 03:58:00.676848 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.657982    4874 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 03:58:00.676848 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.658268    4874 server.go:1014] "Starting client certificate rotation"
Oct 30 03:58:00.676848 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.658391    4874 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 03:58:00.676848 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.658428    4874 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 03:52:20 +0000 UTC" deadline="2028-07-25 18:55:16.166779262 +0000 UTC"
Oct 30 03:58:00.676848 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.658461    4874 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15230h57m15.50832014s"
Oct 30 03:58:00.676848 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.659649    4874 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 03:58:00.676848 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.661371    4874 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 03:58:00.676848 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.662748    4874 log.go:25] "Validated CRI v1 runtime API"
Oct 30 03:58:00.677229 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.664424    4874 log.go:25] "Validated CRI v1 image API"
Oct 30 03:58:00.677229 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.665158    4874 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 03:58:00.677229 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.666751    4874 fs.go:135] Filesystem UUIDs: map[064bcac5-4be3-441f-8b6e-96c55953d0c2:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8da62ed4-7e90-4cf4-9157-dc8a0f041536:/dev/nvme0n1p3]
Oct 30 03:58:00.677340 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.666771    4874 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0121ea1bb37a9e8348390e8ef77e92d2f83e3f7208c8afd915debcfe1131184d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0121ea1bb37a9e8348390e8ef77e92d2f83e3f7208c8afd915debcfe1131184d/userdata/shm major:0 minor:71 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b399c10fd0f25666919e4b322b41999ca07b978930fc36c37ce9e191d6bef95/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b399c10fd0f25666919e4b322b41999ca07b978930fc36c37ce9e191d6bef95/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/538883722ac1cbccca0fcccc546c2ffe23664419cc962d874aeea186fbe66e8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/538883722ac1cbccca0fcccc546c2ffe23664419cc962d874aeea186fbe66e8b/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5523a203c9a2d75d7b7edb12c5dd54ef4c5676cc461ba54e3b38a768577c6ef0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5523a203c9a2d75d7b7edb12c5dd54ef4c5676cc461ba54e3b38a768577c6ef0/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/571ae192b8909614a7260b4ab6add5fc64778ca2eb4e4272773efcab9e70193c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/571ae192b8909614a7260b4ab6add5fc64778ca2eb4e4272773efcab9e70193c/userdata/shm major:0 minor:101 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/68480559d0e9f8cb40e900ede9424e49bbfd02167fcc634ebfa3bdd93b29813d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/68480559d0e9f8cb40e900ede9424e49bbfd02167fcc634ebfa3bdd93b29813d/userdata/shm major:0 minor:274 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/77952d06a35c50393f61276f061dec66cdaa317609434651004514d570532709/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/77952d06a35c50393f61276f061dec66cdaa317609434651004514d570532709/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/78e7047f49c55034afe2ea6bda525cea7236ce655fbab85b43eb933b778551b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/78e7047f49c55034afe2ea6bda525cea7236ce655fbab85b43eb933b778551b1/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/822a1f9a7cf577e4192c455edd2192ac1dcb180d38395f5438a3ff7757c4e19c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/822a1f9a7cf577e4192c455edd2192ac1dcb180d38395f5438a3ff7757c4e19c/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/be470c5342fe3f1ea2b0543ae821eed994fb7a593b4b1faa4da163a0be40341b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/be470c5342fe3f1ea2b0543ae821eed994fb7a593b4b1faa4da163a0be40341b/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c0ddfb16511a887da3f3bd6c6bf63ae5d336361e54089180428a74ab9ca9f6f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c0ddfb16511a887da3f3bd6c6bf63ae5d336361e54089180428a74ab9ca9f6f9/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d598c99ab6258e618d4d6b0dc36296f4f132eb419b6dbb5fbc9e55d3c3d0c01f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d598c99ab6258e618d4d6b0dc36296f4f132eb419b6dbb5fbc9e55d3c3d0c01f/userdata/shm major:0 minor:265 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dcdefb467a535c40a92e290ebf30908bdb38a82eed0d2db9ec8de3b3c454f2da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dcdefb467a535c40a92e290ebf30908bdb38a82eed0d2db9ec8de3b3c454f2da/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3784334f31f6e4aa220ebcc3d15b2ce829cbf96b1454f6d591796fd249ae7c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3784334f31f6e4aa220ebcc3d15b2ce829cbf96b1454f6d591796fd249ae7c8/userdata/shm major:0 minor:259 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e73929c55e45a035187a471a0bd4973ec6348478df05c7b93f9d7b34d69b9327/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e73929c55e45a035187a471a0bd4973ec6348478df05c7b93f9d7b34d69b9327/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e99ebdc945a00b3970de14be54f3eabc56a7bfaed9413e043cc5311a161e1fb2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e99ebdc945a00b3970de14be54f3eabc56a7bfaed9413e043cc5311a161e1fb2/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f4479c621470ed8e7bd2efe500ae1867cd75ab4ba570118cb724f77c5f5b180a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f4479c621470ed8e7bd2efe500ae1867cd75ab4ba570118cb724f77c5f5b180a/userdata/shm major:0 minor:107 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30e672ce-27b7-4997-a887-b834e6a6797d/volumes/kubernetes.io~projected/kube-api-access-5hl75:{mountpoint:/var/lib/kubelet/pods/30e672ce-27b7-4997-a887-b834e6a6797d/volumes/kubernetes.io~projected/kube-api-access-5hl75 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30e672ce-27b7-4997-a887-b834e6a6797d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/30e672ce-27b7-4997-a887-b834e6a6797d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30e672ce-27b7-4997-a887-b834e6a6797d/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/30e672ce-27b7-4997-a887-b834e6a6797d/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e/volumes/kubernetes.io~projected/kube-api-access-grbsb:{mountpoint:/var/lib/kubelet/pods/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e/volumes/kubernetes.io~projected/kube-api-access-grbsb major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/455076cd-b484-44b6-8a1b-b4c3c0dccab8/volumes/kubernetes.io~projected/kube-api-access-zg642:{mountpoint:/var/lib/kubelet/pods/455076cd-b484-44b6-8a1b-b4c3c0dccab8/volumes/kubernetes.io~projected/kube-api-access-zg642 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4dec34b4-fb69-459e-b130-ca1716d23765/volumes/kubernetes.io~projected/kube-api-access-4qr8r:{mountpoint:/var/lib/kubelet/pods/4dec34b4-fb69-459e-b130-ca1716d23765/volumes/kubernetes.io~projected/kube-api-access-4qr8r major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4dec34b4-fb69-459e-b130-ca1716d23765/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/4dec34b4-fb69-459e-b130-ca1716d23765/volumes/kubernetes.io~secret/metrics-certs major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8/volumes/kubernetes.io~projected/kube-api-access-6kq65:{mountpoint:/var/lib/kubelet/pods/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8/volumes/kubernetes.io~projected/kube-api-access-6kq65 major:0 minor:68 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64f1a482-331f-4490-b03f-f41facd80cfe/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/64f1a482-331f-4490-b03f-f41facd80cfe/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64f1a482-331f-4490-b03f-f41facd80cfe/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/64f1a482-331f-4490-b03f-f41facd80cfe/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64f1a482-331f-4490-b03f-f41facd80cfe/volumes/kubernetes.io~projected/kube-api-access-rszkc:{mountpoint:/var/lib/kubelet/pods/64f1a482-331f-4490-b03f-f41facd80cfe/volumes/kubernetes.io~projected/kube-api-access-rszkc major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67ea77e7-74cc-4bc6-ae16-5a5a797f5097/volumes/kubernetes.io~projected/kube-api-access-zfdvg:{mountpoint:/var/lib/kubelet/pods/67ea77e7-74cc-4bc6-ae16-5a5a797f5097/volumes/kubernetes.io~projected/kube-api-access-zfdvg major:0 minor:70 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95dfe3d1-42fa-41b5-a6d3-4774214d54b1/volumes/kubernetes.io~projected/kube-api-access-6msml:{mountpoint:/var/lib/kubelet/pods/95dfe3d1-42fa-41b5-a6d3-4774214d54b1/volumes/kubernetes.io~projected/kube-api-access-6msml major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95dfe3d1-42fa-41b5-a6d3-4774214d54b1/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/95dfe3d1-42fa-41b5-a6d3-4774214d54b1/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97de5e90-d514-4b8c-8917-2e9894978aaa/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/97de5e90-d514-4b8c-8917-2e9894978aaa/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97de5e90-d514-4b8c-8917-2e9894978aaa/volumes/kubernetes.io~projected/kube-api-access-z24jg:{mountpoint:/var/lib/kubelet/pods/97de5e90-d514-4b8c-8917-2e9894978aaa/volumes/kubernetes.io~projected/kube-api-access-z24jg major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97de5e90-d514-4b8c-8917-2e9894978aaa/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/97de5e90-d514-4b8c-8917-2e9894978aaa/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6755b1f-578f-4e75-86df-be3db7533610/volumes/kubernetes.io~projected/kube-api-access-dt7mf:{mountpoint:/var/lib/kubelet/pods/a6755b1f-578f-4e75-86df-be3db7533610/volumes/kubernetes.io~projected/kube-api-access-dt7mf major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6755b1f-578f-4e75-86df-be3db7533610/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/a6755b1f-578f-4e75-86df-be3db7533610/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ade9bd13-6259-48aa-92b2-52f875b960e8/volumes/kubernetes.io~projected/kube-api-access-lvzdn:{mountpoint:/var/lib/kubelet/pods/ade9bd13-6259-48aa-92b2-52f875b960e8/volumes/kubernetes.io~projected/kube-api-access-lvzdn major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bad69b26-9de8-4c2b-b19c-8a96271f5c86/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/bad69b26-9de8-4c2b-b19c-8a96271f5c86/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/baf40c59-c04b-4c27-b080-6c55887c5c4c/volumes/kubernetes.io~projected/kube-api-access-bxh5g:{mountpoint:/var/lib/kubelet/pods/baf40c59-c04b-4c27-b080-6c55887c5c4c/volumes/kubernetes.io~projected/kube-api-access-bxh5g major:0 minor:69 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bee7738d-afa2-431d-ba81-2d14f142902c/volumes/kubernetes.io~projected/kube-api-access-6h268:{mountpoint:/var/lib/kubelet/pods/bee7738d-afa2-431d-ba81-2d14f142902c/volumes/kubernetes.io~projected/kube-api-access-6h268 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bee7738d-afa2-431d-ba81-2d14f142902c/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/bee7738d-afa2-431d-ba81-2d14f142902c/volumes/kubernetes.io~secret/cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5e46634-9969-4d78-bb91-e598e950b1b0/volumes/kubernetes.io~projected/kube-api-access-jwtgk:{mountpoint:/var/lib/kubelet/pods/d5e46634-9969-4d78-bb91-e598e950b1b0/volumes/kubernetes.io~projected/kube-api-access-jwtgk major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e385d23f-6371-4f69-97b1-6a10303c048f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/e385d23f-6371-4f69-97b1-6a10303c048f/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-103:{mountpoint:/var/lib/containers/storage/overlay/caab5b4b253fc896cbda3bb8a11c99faba631dc7f36139b2d26db6ef0c0218f7/merged major:0 minor:103 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/cdb29e00e0727b4e144343ff7eddaa86525e4797f838d2f6fbb410f7b3af1531/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/6b73985f4b270aafd71ac3ba66fef60a8d5f2409ed998d9c2eab4ed2c2dc1e65/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/139c1d84868c18d59f857a20d93d45e58fb46d2563cdd73e02881090bc82bcfd/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-113:{mountpoint:/var/lib/containers/storage/overlay/45778236d5c6d06eb9dccd487ea0183d6d37b6300e23617cc6908dc4fafc4cbf/merged major:0 minor:113 fsType:overlay blockSize:0} overlay_0-121:{mountpoint:/var/lib/containers/storage/overlay/a0afe11cd1c8e7cacc6d57b861c68fd77cd2afac2790751da431a07048660231/merged major:0 minor:121 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/16bba8c2bc1796325b6b74e4a7fc50f2db926f9b22e9ec93446dedc8140ed54b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/68ea704cc40c876fda7d26d30be59c90177e73cf06423d980819b1f34666cbc2/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/506c2440e9ec11a029aa62bd2806dc266723ad1b4b8ddd1b9f48678832280d40/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/84e6f8deed7d13e75b129fc17a7c0bf9a0b3868b31ad2077dcdba383b5e96bc4/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/96c4740a53849bd98b2a27e474494ba914547df5d5c5277dec1c67add7176b36/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/989ad69b545092e5b6b6e94c83cb35e6c138d1c0c99bb9fac1b51578ce3e82ac/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/b977744de42e7c5648a4a0bd23aad84342c0088a1f65a4e2f3d8a2cb4c73b465/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-156:{mountpoint:/var/lib/containers/storage/overlay/53f4aad2cd5e21da4632e8f7c22f4d04ae342391ba3f081714842e53695dc7ab/merged major:0 minor:156 fsType:overlay blockSize:0} overlay_0-158:{mountpoint:/var/lib/containers/storage/overlay/fcb878ac0f3e51c5da14ceb45a52523a8a29ff5e0a51c655a8094ca1bf3fbffd/merged major:0 minor:158 fsType:overlay blockSize:0} overlay_0-169:{mountpoint:/var/lib/containers/storage/overlay/5d134cab6161ddb183cc56ea838b9915632a40ebfd1205c68dc869672ea085f7/merged major:0 minor:169 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/cb0f051c80dacce1be71912b889ddfd28916bc9647b3c4285d16a476147bce3b/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-176:{mountpoint:/var/lib/containers/storage/overlay/8d09af9709846f925c36fee54da1b974ecbeb1241c110e90a25efeeab676ff29/merged major:0 minor:176 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/e0d66da2706a7a941bf6c44bb91d484ebbd1938a22daa8b1049d6a27c556bb2c/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-189:{mountpoint:/var/lib/containers/storage/overlay/6ac05e88bfb128336bb915674dac2f73074ab4a2a4b2512bb2f830a908b94f61/merged major:0 minor:189 fsType:overlay blockSize:0} overlay_0-194:{mountpoint:/var/lib/containers/storage/overlay/61b2b05cdac99b2585a31a82c04bf0637bbbc86e9c8dc7dc9216da91e95e3a46/merged major:0 minor:194 fsType:overlay blockSize:0} overlay_0-199:{mountpoint:/var/lib/containers/storage/overlay/82d5362293903bcc37fd325549d04be82c235861101061fd7a22982d9ce3c210/merged major:0 minor:199 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/37f2ca1b12e836e955e228cc65b149775f2207ddbe43e3a44ed912c4d0083347/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/56c9d0cf038f314121d484dbcb73c2efe3d0cc308d6439fb977b1cfb47adb36e/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/e2d9b59666122402a3a65110a96ccb13a537354d598be8cb616134473b3d91f5/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/98b8cfe83abc9b9db9264fd206643f9fcb3f32ebb1ef37ff47f566128b40ad09/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/68de46b8b5063ccb6ceed559c819f86373a47d16ffa422a9734926503ef1c64c/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/fbd4a887eedf21c005e5f66c0149cac37b4023c2f589ecf31a0d94bade4017cd/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/1016f1b239b9752b544a5d625e747a619bbf8eda720655c7e957b4ba1f3b93fe/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/f266b1af1e64c7986faae86482ba448310e46bfd03de81982ad5d1cab795deb0/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/2f952f76d0606008ccb919bc50662e9ff5e9f2ca194e879a3452aa01a0bea548/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/671a59a6ab5c19dd1864618df9394d829d13d6996ca69d1d7442acdcb8782f9e/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/1299a2de6997d514ba421976a54d0c74b76475c7afa791374251f1bfd127b842/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/b0e7972bf16d4c155a8f175d730225d7d04f9e0256e6a4edd9d59867e62f87c6/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/47e7b5faa72347d6e91de59f8a53234fbfff3e58b2deb2c699c3da473fc0956e/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/18935e57ddbb60383be665d52b5fd09d394ca4201e4934802cd6fd9df94d2b4d/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/fc7513daadcd78f12057972d54a69b7a7ad21ec5125d2516be4a9600ed38f85a/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/4303d370d4de0bcfc8e9a702320530b8607be710014df2b23ebaa63a6e8cf616/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/d7aa1e7a3d607925e402e14aa235aabbdf15d23be8f4253d7f26728a0d3279fe/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/d01e68955be3f92baf8b0b765681eb1a1e5b7da5afc3cce86f884727ef654f1b/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/5bf854f3da4b67768616143d5c9eb656b5c4c7318137db9025668377ddbc40e1/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/83781e4f68e54d13c843d89ba67e483944079388626a1fba89e5a093c385da1b/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/lib/containers/storage/overlay/0b6ec18da4e90d955a802ff152ce66868de1bbb3fc7c09b551b2761b6f46c8a0/merged major:0 minor:77 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/d0bc7f5cf8709d6e2da725198d87e96935059dfaa6f241e162f5f22a52fcff47/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/0efdab852efc1e69e6c43ee86899bd4fd3bdb8357b32a04ca9099446c0ae5616/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/955f1aeef72d04822e5412ae999ac067cac57bfaa9a023047175337bbe23847b/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-87:{mountpoint:/var/lib/containers/storage/overlay/060addc8cc84904af83a33f9c42a434f08a4831140e7656f28e3d0f9429392d9/merged major:0 minor:87 fsType:overlay blockSize:0} overlay_0-91:{mountpoint:/var/lib/containers/storage/overlay/f556709b2b17f68309d7a6e9aa27beb32e15ab838abe927d6aed36d429d21243/merged major:0 minor:91 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/7b2a5ec6c1631e2f3005d081da32f9a02476f3c1a2fad776f9436cc6f0130a17/merged major:0 minor:97 fsType:overlay blockSize:0} overlay_0-99:{mountpoint:/var/lib/containers/storage/overlay/877d49a6664b2209b219917456099fad36737fc936c53d2f3d218e6925ca1e16/merged major:0 minor:99 fsType:overlay blockSize:0}]
Oct 30 03:58:00.677502 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.672862    4874 manager.go:217] Machine: {Timestamp:2026-10-30 03:58:00.672420852 +0000 UTC m=+0.084816417 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100030 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e192808e641a4239ad2a51c10792f SystemUUID:ec2e1928-08e6-41a4-239a-d2a51c10792f BootID:cc0f5889-70a3-43e8-99ca-ce2918b0c4d7 Filesystems:[{Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3784334f31f6e4aa220ebcc3d15b2ce829cbf96b1454f6d591796fd249ae7c8/userdata/shm DeviceMajor:0 DeviceMinor:259 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/64f1a482-331f-4490-b03f-f41facd80cfe/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8/volumes/kubernetes.io~projected/kube-api-access-6kq65 DeviceMajor:0 DeviceMinor:68 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/77952d06a35c50393f61276f061dec66cdaa317609434651004514d570532709/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/822a1f9a7cf577e4192c455edd2192ac1dcb180d38395f5438a3ff7757c4e19c/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/ade9bd13-6259-48aa-92b2-52f875b960e8/volumes/kubernetes.io~projected/kube-api-access-lvzdn DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/455076cd-b484-44b6-8a1b-b4c3c0dccab8/volumes/kubernetes.io~projected/kube-api-access-zg642 DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/64f1a482-331f-4490-b03f-f41facd80cfe/volumes/kubernetes.io~projected/kube-api-access-rszkc DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e/volumes/kubernetes.io~projected/kube-api-access-grbsb DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-77 DeviceMajor:0 DeviceMinor:77 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-113 DeviceMajor:0 DeviceMinor:113 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5523a203c9a2d75d7b7edb12c5dd54ef4c5676cc461ba54e3b38a768577c6ef0/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-176 DeviceMajor:0 DeviceMinor:176 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95dfe3d1-42fa-41b5-a6d3-4774214d54b1/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a6755b1f-578f-4e75-86df-be3db7533610/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bee7738d-afa2-431d-ba81-2d14f142902c/volumes/kubernetes.io~projected/kube-api-access-6h268 DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d598c99ab6258e618d4d6b0dc36296f4f132eb419b6dbb5fbc9e55d3c3d0c01f/userdata/shm DeviceMajor:0 DeviceMinor:265 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e99ebdc945a00b3970de14be54f3eabc56a7bfaed9413e043cc5311a161e1fb2/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/538883722ac1cbccca0fcccc546c2ffe23664419cc962d874aeea186fbe66e8b/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/97de5e90-d514-4b8c-8917-2e9894978aaa/volumes/kubernetes.io~projected/kube-api-access-z24jg DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/571ae192b8909614a7260b4ab6add5fc64778ca2eb4e4272773efcab9e70193c/userdata/shm DeviceMajor:0 DeviceMinor:101 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-169 DeviceMajor:0 DeviceMinor:169 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4dec34b4-fb69-459e-b130-ca1716d23765/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:253 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/97de5e90-d514-4b8c-8917-2e9894978aaa/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4dec34b4-fb69-459e-b130-ca1716d23765/volumes/kubernetes.io~projected/kube-api-access-4qr8r DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/be470c5342fe3f1ea2b0543ae821eed994fb7a593b4b1faa4da163a0be40341b/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/30e672ce-27b7-4997-a887-b834e6a6797d/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-99 DeviceMajor:0 DeviceMinor:99 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/30e672ce-27b7-4997-a887-b834e6a6797d/volumes/kubernetes.io~projected/kube-api-access-5hl75 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0121ea1bb37a9e8348390e8ef77e92d2f83e3f7208c8afd915debcfe1131184d/userdata/shm DeviceMajor:0 DeviceMinor:71 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/64f1a482-331f-4490-b03f-f41facd80cfe/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/30e672ce-27b7-4997-a887-b834e6a6797d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-87 DeviceMajor:0 DeviceMinor:87 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-156 DeviceMajor:0 DeviceMinor:156 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-121 DeviceMajor:0 DeviceMinor:121 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bad69b26-9de8-4c2b-b19c-8a96271f5c86/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:257 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b399c10fd0f25666919e4b322b41999ca07b978930fc36c37ce9e191d6bef95/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/78e7047f49c55034afe2ea6bda525cea7236ce655fbab85b43eb933b778551b1/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-189 DeviceMajor:0 DeviceMinor:189 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5e46634-9969-4d78-bb91-e598e950b1b0/volumes/kubernetes.io~projected/kube-api-access-jwtgk DeviceMajor:0 DeviceMinor:258 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f4479c621470ed8e7bd2efe500ae1867cd75ab4ba570118cb724f77c5f5b180a/userdata/shm DeviceMajor:0 DeviceMinor:107 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/97de5e90-d514-4b8c-8917-2e9894978aaa/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/a6755b1f-578f-4e75-86df-be3db7533610/volumes/kubernetes.io~projected/kube-api-access-dt7mf DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/95dfe3d1-42fa-41b5-a6d3-4774214d54b1/volumes/kubernetes.io~projected/kube-api-access-6msml DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e73929c55e45a035187a471a0bd4973ec6348478df05c7b93f9d7b34d69b9327/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/baf40c59-c04b-4c27-b080-6c55887c5c4c/volumes/kubernetes.io~projected/kube-api-access-bxh5g DeviceMajor:0 DeviceMinor:69 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-91 DeviceMajor:0 DeviceMinor:91 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-103 DeviceMajor:0 DeviceMinor:103 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-158 DeviceMajor:0 DeviceMinor:158 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-199 DeviceMajor:0 DeviceMinor:199 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67ea77e7-74cc-4bc6-ae16-5a5a797f5097/volumes/kubernetes.io~projected/kube-api-access-zfdvg DeviceMajor:0 DeviceMinor:70 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dcdefb467a535c40a92e290ebf30908bdb38a82eed0d2db9ec8de3b3c454f2da/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e385d23f-6371-4f69-97b1-6a10303c048f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c0ddfb16511a887da3f3bd6c6bf63ae5d336361e54089180428a74ab9ca9f6f9/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-194 DeviceMajor:0 DeviceMinor:194 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bee7738d-afa2-431d-ba81-2d14f142902c/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/68480559d0e9f8cb40e900ede9424e49bbfd02167fcc634ebfa3bdd93b29813d/userdata/shm DeviceMajor:0 DeviceMinor:274 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:538883722ac1cbc MacAddress:1e:5e:77:d2:e1:99 Speed:10000 Mtu:8901} {Name:68480559d0e9f8c MacAddress:da:1a:b8:ab:70:c3 Speed:10000 Mtu:8901} {Name:822a1f9a7cf577e MacAddress:96:65:03:17:5f:95 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d8:7a:70:9f Speed:0 Mtu:9001} {Name:br-int MacAddress:02:5f:fb:83:ef:0a Speed:0 Mtu:8901} {Name:d598c99ab6258e6 MacAddress:5e:dc:99:48:34:77 Speed:10000 Mtu:8901} {Name:e3784334f31f6e4 MacAddress:32:71:00:76:f3:db Speed:10000 Mtu:8901} {Name:e73929c55e45a03 MacAddress:b6:98:72:56:fc:4a Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d8:7a:70:9f Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:92:a7:80:73:9d:c2 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:16:04:d3:70:e1:b2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 03:58:00.677502 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.673124    4874 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 03:58:00.677502 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.673162    4874 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 03:58:00.677502 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.673431    4874 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 03:58:00.677502 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.673450    4874 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-160.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 03:58:00.677502 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.673566    4874 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 03:58:00.677502 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.673573    4874 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 03:58:00.677502 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.673586    4874 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 03:58:00.677502 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.673605    4874 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 03:58:00.677502 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.673679    4874 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 03:58:00.677502 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.673821    4874 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 03:58:00.677502 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.673918    4874 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 03:58:00.677502 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.673954    4874 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 03:58:00.677502 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.673967    4874 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 03:58:00.677502 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.673976    4874 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 03:58:00.677502 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.673985    4874 kubelet.go:405] "Adding apiserver pod source"
Oct 30 03:58:00.677502 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.673992    4874 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 03:58:00.677502 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.675437    4874 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 03:58:00.677502 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.675457    4874 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 03:58:00.677502 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.676515    4874 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 03:58:00.679409 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.679393    4874 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 03:58:00.679447 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.679412    4874 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 03:58:00.679535 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.679525    4874 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 03:58:00.679708 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.679698    4874 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 03:58:00.679738 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.679727    4874 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 03:58:00.679826 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.679817    4874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 03:58:00.679863 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.679831    4874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 03:58:00.679863 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.679838    4874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 03:58:00.679863 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.679844    4874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 03:58:00.679863 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.679849    4874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 03:58:00.679863 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.679855    4874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 03:58:00.679863 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.679863    4874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 03:58:00.680016 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.679871    4874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 03:58:00.680016 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.679878    4874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 03:58:00.680016 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.679884    4874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 03:58:00.680016 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.679892    4874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 03:58:00.680016 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.679901    4874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 03:58:00.680016 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.679925    4874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 03:58:00.680016 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.679944    4874 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 03:58:00.683092 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.683078    4874 server.go:1263] "Started kubelet"
Oct 30 03:58:00.683517 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.683156    4874 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 03:58:00.683517 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.683262    4874 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 03:58:00.683517 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.683334    4874 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 03:58:00.683708 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.683609    4874 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 03:58:00.683596 ip-10-0-138-160 systemd[1]: Started Kubernetes Kubelet.
Oct 30 03:58:00.684532 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.684516    4874 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 03:58:00.687091 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.687071    4874 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 03:58:00.688630 ip-10-0-138-160 kubenswrapper[4874]: E1030 03:58:00.688335    4874 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-160.ec2.internal\" not found"
Oct 30 03:58:00.688630 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.688396    4874 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 03:58:00.688630 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.688404    4874 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 03:58:00.688630 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.688508    4874 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 03:58:00.688630 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.688585    4874 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 03:58:00.688949 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.688672    4874 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 03:52:22 +0000 UTC" deadline="2028-08-08 19:42:28.728523184 +0000 UTC"
Oct 30 03:58:00.688949 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.688705    4874 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15567h44m28.039821129s"
Oct 30 03:58:00.689825 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.689760    4874 factory.go:55] Registering systemd factory
Oct 30 03:58:00.689825 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.689784    4874 factory.go:223] Registration of the systemd container factory successfully
Oct 30 03:58:00.691469 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.690570    4874 factory.go:153] Registering CRI-O factory
Oct 30 03:58:00.691469 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.690585    4874 factory.go:223] Registration of the crio container factory successfully
Oct 30 03:58:00.691469 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.690664    4874 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 03:58:00.691469 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.690682    4874 factory.go:103] Registering Raw factory
Oct 30 03:58:00.691469 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.690697    4874 manager.go:1199] Started watching for new ooms in manager
Oct 30 03:58:00.691469 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691024    4874 manager.go:319] Starting recovery of all containers
Oct 30 03:58:00.691469 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691146    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97de5e90-d514-4b8c-8917-2e9894978aaa" volumeName="kubernetes.io/secret/97de5e90-d514-4b8c-8917-2e9894978aaa-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 03:58:00.691469 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691187    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6755b1f-578f-4e75-86df-be3db7533610" volumeName="kubernetes.io/secret/a6755b1f-578f-4e75-86df-be3db7533610-metrics-tls" seLinuxMountContext=""
Oct 30 03:58:00.691469 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691206    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bee7738d-afa2-431d-ba81-2d14f142902c" volumeName="kubernetes.io/secret/bee7738d-afa2-431d-ba81-2d14f142902c-cert" seLinuxMountContext=""
Oct 30 03:58:00.691469 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691220    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30e672ce-27b7-4997-a887-b834e6a6797d" volumeName="kubernetes.io/empty-dir/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-textfile" seLinuxMountContext=""
Oct 30 03:58:00.691469 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691259    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30e672ce-27b7-4997-a887-b834e6a6797d" volumeName="kubernetes.io/secret/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-tls" seLinuxMountContext=""
Oct 30 03:58:00.691469 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691274    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c9f3a00-e77c-4946-b5bb-3a0cb171b83e" volumeName="kubernetes.io/projected/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-kube-api-access-grbsb" seLinuxMountContext=""
Oct 30 03:58:00.691469 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691292    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dec34b4-fb69-459e-b130-ca1716d23765" volumeName="kubernetes.io/secret/4dec34b4-fb69-459e-b130-ca1716d23765-metrics-certs" seLinuxMountContext=""
Oct 30 03:58:00.691469 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691306    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67ea77e7-74cc-4bc6-ae16-5a5a797f5097" volumeName="kubernetes.io/configmap/67ea77e7-74cc-4bc6-ae16-5a5a797f5097-iptables-alerter-script" seLinuxMountContext=""
Oct 30 03:58:00.691469 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691328    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97de5e90-d514-4b8c-8917-2e9894978aaa" volumeName="kubernetes.io/projected/97de5e90-d514-4b8c-8917-2e9894978aaa-kube-api-access-z24jg" seLinuxMountContext=""
Oct 30 03:58:00.691469 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691342    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baf40c59-c04b-4c27-b080-6c55887c5c4c" volumeName="kubernetes.io/projected/baf40c59-c04b-4c27-b080-6c55887c5c4c-kube-api-access-bxh5g" seLinuxMountContext=""
Oct 30 03:58:00.691469 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691369    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e385d23f-6371-4f69-97b1-6a10303c048f" volumeName="kubernetes.io/secret/e385d23f-6371-4f69-97b1-6a10303c048f-agent-certs" seLinuxMountContext=""
Oct 30 03:58:00.691469 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691389    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67ea77e7-74cc-4bc6-ae16-5a5a797f5097" volumeName="kubernetes.io/projected/67ea77e7-74cc-4bc6-ae16-5a5a797f5097-kube-api-access-zfdvg" seLinuxMountContext=""
Oct 30 03:58:00.691469 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691403    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97de5e90-d514-4b8c-8917-2e9894978aaa" volumeName="kubernetes.io/configmap/97de5e90-d514-4b8c-8917-2e9894978aaa-env-overrides" seLinuxMountContext=""
Oct 30 03:58:00.691469 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691423    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97de5e90-d514-4b8c-8917-2e9894978aaa" volumeName="kubernetes.io/configmap/97de5e90-d514-4b8c-8917-2e9894978aaa-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 03:58:00.691469 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691441    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e385d23f-6371-4f69-97b1-6a10303c048f" volumeName="kubernetes.io/configmap/e385d23f-6371-4f69-97b1-6a10303c048f-konnectivity-ca" seLinuxMountContext=""
Oct 30 03:58:00.691469 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691460    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="455076cd-b484-44b6-8a1b-b4c3c0dccab8" volumeName="kubernetes.io/projected/455076cd-b484-44b6-8a1b-b4c3c0dccab8-kube-api-access-zg642" seLinuxMountContext=""
Oct 30 03:58:00.691469 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691473    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dec34b4-fb69-459e-b130-ca1716d23765" volumeName="kubernetes.io/projected/4dec34b4-fb69-459e-b130-ca1716d23765-kube-api-access-4qr8r" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691486    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95dfe3d1-42fa-41b5-a6d3-4774214d54b1" volumeName="kubernetes.io/empty-dir/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-data-volume" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691504    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95dfe3d1-42fa-41b5-a6d3-4774214d54b1" volumeName="kubernetes.io/projected/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-kube-api-access-6msml" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691517    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ade9bd13-6259-48aa-92b2-52f875b960e8" volumeName="kubernetes.io/projected/ade9bd13-6259-48aa-92b2-52f875b960e8-kube-api-access-lvzdn" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691535    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bad69b26-9de8-4c2b-b19c-8a96271f5c86" volumeName="kubernetes.io/secret/bad69b26-9de8-4c2b-b19c-8a96271f5c86-original-pull-secret" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691549    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30e672ce-27b7-4997-a887-b834e6a6797d" volumeName="kubernetes.io/configmap/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691562    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30e672ce-27b7-4997-a887-b834e6a6797d" volumeName="kubernetes.io/projected/30e672ce-27b7-4997-a887-b834e6a6797d-kube-api-access-5hl75" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691580    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c9f3a00-e77c-4946-b5bb-3a0cb171b83e" volumeName="kubernetes.io/configmap/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-cni-binary-copy" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691593    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8" volumeName="kubernetes.io/configmap/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691611    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8" volumeName="kubernetes.io/projected/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-kube-api-access-6kq65" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691627    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64f1a482-331f-4490-b03f-f41facd80cfe" volumeName="kubernetes.io/empty-dir/64f1a482-331f-4490-b03f-f41facd80cfe-etc-tuned" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691640    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64f1a482-331f-4490-b03f-f41facd80cfe" volumeName="kubernetes.io/projected/64f1a482-331f-4490-b03f-f41facd80cfe-kube-api-access-rszkc" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691668    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6755b1f-578f-4e75-86df-be3db7533610" volumeName="kubernetes.io/projected/a6755b1f-578f-4e75-86df-be3db7533610-kube-api-access-dt7mf" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691689    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c9f3a00-e77c-4946-b5bb-3a0cb171b83e" volumeName="kubernetes.io/configmap/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-multus-daemon-config" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691703    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8" volumeName="kubernetes.io/configmap/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691721    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95dfe3d1-42fa-41b5-a6d3-4774214d54b1" volumeName="kubernetes.io/secret/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691848    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ade9bd13-6259-48aa-92b2-52f875b960e8" volumeName="kubernetes.io/empty-dir/ade9bd13-6259-48aa-92b2-52f875b960e8-tmp-dir" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691879    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30e672ce-27b7-4997-a887-b834e6a6797d" volumeName="kubernetes.io/configmap/30e672ce-27b7-4997-a887-b834e6a6797d-metrics-client-ca" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691894    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30e672ce-27b7-4997-a887-b834e6a6797d" volumeName="kubernetes.io/secret/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691903    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64f1a482-331f-4490-b03f-f41facd80cfe" volumeName="kubernetes.io/empty-dir/64f1a482-331f-4490-b03f-f41facd80cfe-tmp" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691914    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6755b1f-578f-4e75-86df-be3db7533610" volumeName="kubernetes.io/configmap/a6755b1f-578f-4e75-86df-be3db7533610-config-volume" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691925    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6755b1f-578f-4e75-86df-be3db7533610" volumeName="kubernetes.io/empty-dir/a6755b1f-578f-4e75-86df-be3db7533610-tmp-dir" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691947    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bee7738d-afa2-431d-ba81-2d14f142902c" volumeName="kubernetes.io/projected/bee7738d-afa2-431d-ba81-2d14f142902c-kube-api-access-6h268" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691965    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5e46634-9969-4d78-bb91-e598e950b1b0" volumeName="kubernetes.io/projected/d5e46634-9969-4d78-bb91-e598e950b1b0-kube-api-access-jwtgk" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691978    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="455076cd-b484-44b6-8a1b-b4c3c0dccab8" volumeName="kubernetes.io/configmap/455076cd-b484-44b6-8a1b-b4c3c0dccab8-serviceca" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.691995    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8" volumeName="kubernetes.io/configmap/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-cni-binary-copy" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.692013    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95dfe3d1-42fa-41b5-a6d3-4774214d54b1" volumeName="kubernetes.io/configmap/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.692024    4874 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97de5e90-d514-4b8c-8917-2e9894978aaa" volumeName="kubernetes.io/configmap/97de5e90-d514-4b8c-8917-2e9894978aaa-ovnkube-config" seLinuxMountContext=""
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.692034    4874 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 03:58:00.692568 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.692041    4874 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 03:58:00.704498 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.704448    4874 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 03:58:00.715426 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.715394    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 03:58:00.715602 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.715579    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 03:58:00.721298 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.721272    4874 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 03:58:00.721298 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.721296    4874 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 03:58:00.721458 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.721312    4874 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 03:58:00.721458 ip-10-0-138-160 kubenswrapper[4874]: E1030 03:58:00.721353    4874 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 03:58:00.729041 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.729002    4874 generic.go:356] "Generic (PLEG): container finished" podID="5bcce5d87346fa9cc82e9cdb4fa012bd" containerID="272526d88d54c2ab15fed2187134833e4c41d4abed1ebf882a8abc7e519cc400" exitCode=0
Oct 30 03:58:00.734111 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.734091    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 03:58:00.734449 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.734424    4874 generic.go:356] "Generic (PLEG): container finished" podID="97de5e90-d514-4b8c-8917-2e9894978aaa" containerID="5714c57cd1973935027280deb7dededf26ab4da87ba0ae006351458ecb0f6bc8" exitCode=1
Oct 30 03:58:00.734927 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.734902    4874 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:58:00.735008 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.734943    4874 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 03:58:00.735396 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.735380    4874 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:58:00.744275 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.744225    4874 generic.go:356] "Generic (PLEG): container finished" podID="30e672ce-27b7-4997-a887-b834e6a6797d" containerID="d2ff697079aee63dc22b2f07a49c242950d8b13e74a699ffd9151e2f4893c218" exitCode=0
Oct 30 03:58:00.757336 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.757318    4874 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:58:00.757526 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.757506    4874 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:58:00.762584 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.762564    4874 generic.go:356] "Generic (PLEG): container finished" podID="4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8" containerID="f449f312f714a685ea8bd0cdb6a12904911d9d08af814de7b93e4085bbebbac6" exitCode=0
Oct 30 03:58:00.762584 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.762582    4874 generic.go:356] "Generic (PLEG): container finished" podID="4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8" containerID="d6c5b21006b70ee9735297284090399a087d1a2dc8fd743a61bb14bc971a795c" exitCode=0
Oct 30 03:58:00.762692 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.762588    4874 generic.go:356] "Generic (PLEG): container finished" podID="4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8" containerID="fca7af361ec57381ea41e64351e6a33fb35eb6a2da611435fab3c75245dc4350" exitCode=0
Oct 30 03:58:00.762692 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.762596    4874 generic.go:356] "Generic (PLEG): container finished" podID="4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8" containerID="f36e067f9b4019ff1a3a82fe38b077e7b124f4710d92241f4ccec47e218bf92f" exitCode=0
Oct 30 03:58:00.762692 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.762601    4874 generic.go:356] "Generic (PLEG): container finished" podID="4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8" containerID="1af63fe35fccea7a7c1ada89683a1c6d92a961ac01113f8ada8653bf9e6321bb" exitCode=0
Oct 30 03:58:00.762692 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.762606    4874 generic.go:356] "Generic (PLEG): container finished" podID="4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8" containerID="d667734d25643e1d838954f2f2020f4c49c12d1e92ab9906b79d6e12edcfa53e" exitCode=0
Oct 30 03:58:00.768669 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.768654    4874 manager.go:324] Recovery completed
Oct 30 03:58:00.786074 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.786026    4874 cpu_manager.go:226] "Starting" policy="none"
Oct 30 03:58:00.786074 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.786038    4874 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 03:58:00.786074 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.786053    4874 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 03:58:00.786203 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.786155    4874 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 03:58:00.786203 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.786165    4874 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 03:58:00.786203 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.786191    4874 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 03:58:00.786203 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.786197    4874 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 03:58:00.786203 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.786203    4874 policy_none.go:50] "Start"
Oct 30 03:58:00.786353 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.786211    4874 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 03:58:00.786353 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.786220    4874 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 03:58:00.786403 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.786352    4874 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 03:58:00.786403 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.786360    4874 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 03:58:00.786403 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.786365    4874 policy_none.go:44] "Start"
Oct 30 03:58:00.790379 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.790353    4874 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 03:58:00.794719 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.794706    4874 manager.go:341] "Starting Device Plugin manager"
Oct 30 03:58:00.795145 ip-10-0-138-160 kubenswrapper[4874]: E1030 03:58:00.794730    4874 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 03:58:00.795188 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.795147    4874 server.go:87] "Starting device plugin registration server"
Oct 30 03:58:00.795404 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.795394    4874 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 03:58:00.795436 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.795405    4874 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 03:58:00.795531 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.795509    4874 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 03:58:00.795740 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.795675    4874 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 03:58:00.795852 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.795746    4874 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 03:58:00.795852 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.795754    4874 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 03:58:00.795961 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.795823    4874 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T03:58:00.795734479Z","UUID":"9472565a-79ac-4999-828a-70e0bae36d83","Handler":null,"Name":"","Endpoint":""}
Oct 30 03:58:00.796934 ip-10-0-138-160 kubenswrapper[4874]: E1030 03:58:00.796918    4874 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 03:58:00.797148 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.797135    4874 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 03:58:00.797203 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.797163    4874 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 03:58:00.822012 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.821978    4874 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-160.ec2.internal"]
Oct 30 03:58:00.822566 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.822530    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal" event={"ID":"5bcce5d87346fa9cc82e9cdb4fa012bd","Type":"ContainerStarted","Data":"be470c5342fe3f1ea2b0543ae821eed994fb7a593b4b1faa4da163a0be40341b"}
Oct 30 03:58:00.822614 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.822565    4874 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal"
Oct 30 03:58:00.822644 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.822627    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-160.ec2.internal" event={"ID":"bc4b060adf5b640eabc59287108542f2","Type":"ContainerStarted","Data":"82d1a5d9fe6c5d4342a4dad2903f43d431c1b1f52cde6480ebfcc67c16cb3de6"}
Oct 30 03:58:00.822644 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.822637    4874 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-160.ec2.internal"
Oct 30 03:58:00.822714 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.822643    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-160.ec2.internal" event={"ID":"bc4b060adf5b640eabc59287108542f2","Type":"ContainerStarted","Data":"dcdefb467a535c40a92e290ebf30908bdb38a82eed0d2db9ec8de3b3c454f2da"}
Oct 30 03:58:00.845450 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.845431    4874 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 03:58:00.845528 ip-10-0-138-160 kubenswrapper[4874]: E1030 03:58:00.845467    4874 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-160.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-160.ec2.internal"
Oct 30 03:58:00.846167 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.846152    4874 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 03:58:00.846217 ip-10-0-138-160 kubenswrapper[4874]: E1030 03:58:00.846198    4874 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal"
Oct 30 03:58:00.896501 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.896484    4874 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:58:00.897583 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.897571    4874 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:58:00.897640 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.897594    4874 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:58:00.897640 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.897603    4874 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-160.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:58:00.897640 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.897638    4874 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-160.ec2.internal"
Oct 30 03:58:00.931840 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.931824    4874 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-160.ec2.internal"
Oct 30 03:58:00.931898 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.931872    4874 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-160.ec2.internal"
Oct 30 03:58:00.936450 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.936434    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5bcce5d87346fa9cc82e9cdb4fa012bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal\" (UID: \"5bcce5d87346fa9cc82e9cdb4fa012bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal"
Oct 30 03:58:00.936522 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.936458    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bcce5d87346fa9cc82e9cdb4fa012bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal\" (UID: \"5bcce5d87346fa9cc82e9cdb4fa012bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal"
Oct 30 03:58:00.936522 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:00.936473    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc4b060adf5b640eabc59287108542f2-config\") pod \"kube-apiserver-proxy-ip-10-0-138-160.ec2.internal\" (UID: \"bc4b060adf5b640eabc59287108542f2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-160.ec2.internal"
Oct 30 03:58:01.037160 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.037113    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5bcce5d87346fa9cc82e9cdb4fa012bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal\" (UID: \"5bcce5d87346fa9cc82e9cdb4fa012bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal"
Oct 30 03:58:01.037160 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.037074    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5bcce5d87346fa9cc82e9cdb4fa012bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal\" (UID: \"5bcce5d87346fa9cc82e9cdb4fa012bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal"
Oct 30 03:58:01.037277 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.037173    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bcce5d87346fa9cc82e9cdb4fa012bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal\" (UID: \"5bcce5d87346fa9cc82e9cdb4fa012bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal"
Oct 30 03:58:01.037277 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.037190    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc4b060adf5b640eabc59287108542f2-config\") pod \"kube-apiserver-proxy-ip-10-0-138-160.ec2.internal\" (UID: \"bc4b060adf5b640eabc59287108542f2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-160.ec2.internal"
Oct 30 03:58:01.037277 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.037223    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc4b060adf5b640eabc59287108542f2-config\") pod \"kube-apiserver-proxy-ip-10-0-138-160.ec2.internal\" (UID: \"bc4b060adf5b640eabc59287108542f2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-160.ec2.internal"
Oct 30 03:58:01.037277 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.037267    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bcce5d87346fa9cc82e9cdb4fa012bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal\" (UID: \"5bcce5d87346fa9cc82e9cdb4fa012bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal"
Oct 30 03:58:01.674465 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.674431    4874 apiserver.go:52] "Watching apiserver"
Oct 30 03:58:01.726132 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.726106    4874 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 03:58:01.726576 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.726554    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-9nxvt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal","openshift-cluster-node-tuning-operator/tuned-b88ss","openshift-ingress-canary/ingress-canary-2ldpw","openshift-network-operator/iptables-alerter-ttsr6","openshift-monitoring/node-exporter-dgvb4","kube-system/konnectivity-agent-vnv9w","openshift-image-registry/node-ca-lrcg5","openshift-network-diagnostics/network-check-target-9mktl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4","openshift-dns/dns-default-mctfq","openshift-dns/node-resolver-8g477","openshift-insights/insights-runtime-extractor-8qhbh","openshift-multus/multus-zl7qk","kube-system/global-pull-secret-syncer-cbfm8","openshift-ovn-kubernetes/ovnkube-node-244hd","openshift-multus/network-metrics-daemon-dpdr4","kube-system/kube-apiserver-proxy-ip-10-0-138-160.ec2.internal"]
Oct 30 03:58:01.741140 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.741115    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-vnctk\""
Oct 30 03:58:01.741254 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.741152    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-skmdf\""
Oct 30 03:58:01.741317 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.741301    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 03:58:01.741365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.741351    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 03:58:01.741395 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.741373    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:58:01.741427 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.741404    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 03:58:01.741455 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.741443    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 03:58:01.741482 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.741470    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:58:01.741509 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.741502    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 03:58:01.752208 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.752184    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 03:58:01.752854 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.752835    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 03:58:01.753125 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.753108    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-smvt7\""
Oct 30 03:58:01.753125 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.753119    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-dnvhh\""
Oct 30 03:58:01.753232 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.753122    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vm8lh\""
Oct 30 03:58:01.753968 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.753906    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4xznk\""
Oct 30 03:58:01.753968 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.753905    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 03:58:01.753968 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.753956    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h7p9v\""
Oct 30 03:58:01.753968 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.753967    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 03:58:01.754224 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.753959    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rs54w\""
Oct 30 03:58:01.754224 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.754026    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 03:58:01.754224 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.754165    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kwf82\""
Oct 30 03:58:01.754224 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.754209    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hmk7j\""
Oct 30 03:58:01.754420 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.754341    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 03:58:01.754420 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.754344    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 03:58:01.754685 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.754670    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mwxht\""
Oct 30 03:58:01.754685 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.754690    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 03:58:01.754870 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.754851    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:58:01.754914 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.754873    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 03:58:01.755044 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.755021    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8s8mj\""
Oct 30 03:58:01.755142 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.755119    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b2qtm\""
Oct 30 03:58:01.755235 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.755156    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 03:58:01.755235 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.755174    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 03:58:01.755235 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.755201    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 03:58:01.755235 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.755206    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 03:58:01.755235 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.755176    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 03:58:01.755442 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.755261    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 03:58:01.755442 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.755272    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 03:58:01.755442 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.755264    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 03:58:01.755442 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.755284    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 03:58:01.755442 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.755311    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 03:58:01.755442 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.755313    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 03:58:01.755442 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.755324    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 03:58:01.763447 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.763432    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 03:58:01.763755 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.763741    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 03:58:01.763812 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.763798    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 03:58:01.766446 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.764942    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 03:58:01.766446 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.764993    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 03:58:01.766446 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.765026    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 03:58:01.766446 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.765052    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 03:58:01.766446 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.765071    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 03:58:01.766446 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.765098    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 03:58:01.766446 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.765129    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w4blj\""
Oct 30 03:58:01.766446 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.765144    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j9mnz\""
Oct 30 03:58:01.766446 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.765146    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 03:58:01.766446 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.765184    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 03:58:01.766446 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.765225    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-846td\""
Oct 30 03:58:01.766446 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.765296    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 03:58:01.766446 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.765428    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 03:58:01.766446 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.765441    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 03:58:01.766446 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.765458    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 03:58:01.766446 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.765491    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:58:01.766446 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.765617    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 03:58:01.766446 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.766193    4874 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-160.ec2.internal"
Oct 30 03:58:01.767095 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.766629    4874 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal"
Oct 30 03:58:01.787007 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.786989    4874 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 03:58:01.787105 ip-10-0-138-160 kubenswrapper[4874]: E1030 03:58:01.787037    4874 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-160.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-160.ec2.internal"
Oct 30 03:58:01.787774 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.787763    4874 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 03:58:01.787819 ip-10-0-138-160 kubenswrapper[4874]: E1030 03:58:01.787788    4874 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-160.ec2.internal"
Oct 30 03:58:01.789682 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.789669    4874 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 03:58:01.840820 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.840796    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-hostroot\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.840969 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.840825    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:58:01.840969 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.840844    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfdvg\" (UniqueName: \"kubernetes.io/projected/67ea77e7-74cc-4bc6-ae16-5a5a797f5097-kube-api-access-zfdvg\") pod \"iptables-alerter-ttsr6\" (UID: \"67ea77e7-74cc-4bc6-ae16-5a5a797f5097\") " pod="openshift-network-operator/iptables-alerter-ttsr6"
Oct 30 03:58:01.840969 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.840859    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-var-lib-kubelet\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.840969 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.840936    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-systemd-units\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.841123 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.841104    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.841157 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.841133    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-var-lib-cni-multus\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.841198 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.841163    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30e672ce-27b7-4997-a887-b834e6a6797d-root\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:58:01.841233 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.841196    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-tls\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:58:01.841233 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.841229    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e385d23f-6371-4f69-97b1-6a10303c048f-konnectivity-ca\") pod \"konnectivity-agent-vnv9w\" (UID: \"e385d23f-6371-4f69-97b1-6a10303c048f\") " pod="kube-system/konnectivity-agent-vnv9w"
Oct 30 03:58:01.841322 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.841264    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg642\" (UniqueName: \"kubernetes.io/projected/455076cd-b484-44b6-8a1b-b4c3c0dccab8-kube-api-access-zg642\") pod \"node-ca-lrcg5\" (UID: \"455076cd-b484-44b6-8a1b-b4c3c0dccab8\") " pod="openshift-image-registry/node-ca-lrcg5"
Oct 30 03:58:01.841416 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.841395    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwtgk\" (UniqueName: \"kubernetes.io/projected/d5e46634-9969-4d78-bb91-e598e950b1b0-kube-api-access-jwtgk\") pod \"network-check-target-9mktl\" (UID: \"d5e46634-9969-4d78-bb91-e598e950b1b0\") " pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:58:01.841463 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.841446    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e385d23f-6371-4f69-97b1-6a10303c048f-konnectivity-ca\") pod \"konnectivity-agent-vnv9w\" (UID: \"e385d23f-6371-4f69-97b1-6a10303c048f\") " pod="kube-system/konnectivity-agent-vnv9w"
Oct 30 03:58:01.841511 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.841470    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-run\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.841511 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.841501    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-host\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.841595 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.841533    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/97de5e90-d514-4b8c-8917-2e9894978aaa-ovnkube-script-lib\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.841640 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.841618    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-multus-cni-dir\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.841696 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.841673    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bad69b26-9de8-4c2b-b19c-8a96271f5c86-kubelet-config\") pod \"global-pull-secret-syncer-cbfm8\" (UID: \"bad69b26-9de8-4c2b-b19c-8a96271f5c86\") " pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:58:01.841725 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.841713    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-tls\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:58:01.841753 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.841721    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:58:01.841753 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.841743    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6755b1f-578f-4e75-86df-be3db7533610-tmp-dir\") pod \"dns-default-mctfq\" (UID: \"a6755b1f-578f-4e75-86df-be3db7533610\") " pod="openshift-dns/dns-default-mctfq"
Oct 30 03:58:01.841821 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.841810    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6755b1f-578f-4e75-86df-be3db7533610-tmp-dir\") pod \"dns-default-mctfq\" (UID: \"a6755b1f-578f-4e75-86df-be3db7533610\") " pod="openshift-dns/dns-default-mctfq"
Oct 30 03:58:01.841879 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.841861    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxh5g\" (UniqueName: \"kubernetes.io/projected/baf40c59-c04b-4c27-b080-6c55887c5c4c-kube-api-access-bxh5g\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:58:01.841909 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.841892    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:58:01.841949 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.841895    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-system-cni-dir\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.842025 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.841947    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-run-netns\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.842025 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.841989    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bad69b26-9de8-4c2b-b19c-8a96271f5c86-dbus\") pod \"global-pull-secret-syncer-cbfm8\" (UID: \"bad69b26-9de8-4c2b-b19c-8a96271f5c86\") " pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:58:01.842025 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.841995    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/97de5e90-d514-4b8c-8917-2e9894978aaa-ovnkube-script-lib\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.842118 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842025    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kq65\" (UniqueName: \"kubernetes.io/projected/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-kube-api-access-6kq65\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:58:01.842118 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842077    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-registration-dir\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:58:01.842180 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842165    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-slash\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.842255 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842224    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-var-lib-openvswitch\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.842289 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842269    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-run-openvswitch\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.842319 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842286    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-log-socket\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.842319 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842301    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-wtmp\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:58:01.842319 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842315    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-system-cni-dir\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:58:01.842446 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842329    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-crio-socket\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:58:01.842446 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842343    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-sysctl-conf\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.842446 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842380    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-var-lib-cni-bin\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.842446 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842426    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-etc-kubernetes\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.842637 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842455    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-textfile\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:58:01.842637 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842511    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ade9bd13-6259-48aa-92b2-52f875b960e8-hosts-file\") pod \"node-resolver-8g477\" (UID: \"ade9bd13-6259-48aa-92b2-52f875b960e8\") " pod="openshift-dns/node-resolver-8g477"
Oct 30 03:58:01.842637 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842540    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dec34b4-fb69-459e-b130-ca1716d23765-metrics-certs\") pod \"network-metrics-daemon-dpdr4\" (UID: \"4dec34b4-fb69-459e-b130-ca1716d23765\") " pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:58:01.842637 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842538    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-textfile\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:58:01.842637 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842568    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-device-dir\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:58:01.842861 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842650    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-sys-fs\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:58:01.842861 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842674    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-kubernetes\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.842861 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842690    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-node-log\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.842861 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842731    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-run-k8s-cni-cncf-io\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.842861 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842740    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dec34b4-fb69-459e-b130-ca1716d23765-metrics-certs\") pod \"network-metrics-daemon-dpdr4\" (UID: \"4dec34b4-fb69-459e-b130-ca1716d23765\") " pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:58:01.842861 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842764    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/455076cd-b484-44b6-8a1b-b4c3c0dccab8-host\") pod \"node-ca-lrcg5\" (UID: \"455076cd-b484-44b6-8a1b-b4c3c0dccab8\") " pod="openshift-image-registry/node-ca-lrcg5"
Oct 30 03:58:01.842861 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842786    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:58:01.842861 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842824    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-etc-openvswitch\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.843219 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842878    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-cni-netd\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.843219 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842896    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-cni-binary-copy\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.843219 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842911    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hl75\" (UniqueName: \"kubernetes.io/projected/30e672ce-27b7-4997-a887-b834e6a6797d-kube-api-access-5hl75\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:58:01.843219 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842926    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade9bd13-6259-48aa-92b2-52f875b960e8-tmp-dir\") pod \"node-resolver-8g477\" (UID: \"ade9bd13-6259-48aa-92b2-52f875b960e8\") " pod="openshift-dns/node-resolver-8g477"
Oct 30 03:58:01.843219 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842942    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvzdn\" (UniqueName: \"kubernetes.io/projected/ade9bd13-6259-48aa-92b2-52f875b960e8-kube-api-access-lvzdn\") pod \"node-resolver-8g477\" (UID: \"ade9bd13-6259-48aa-92b2-52f875b960e8\") " pod="openshift-dns/node-resolver-8g477"
Oct 30 03:58:01.843219 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.842983    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt7mf\" (UniqueName: \"kubernetes.io/projected/a6755b1f-578f-4e75-86df-be3db7533610-kube-api-access-dt7mf\") pod \"dns-default-mctfq\" (UID: \"a6755b1f-578f-4e75-86df-be3db7533610\") " pod="openshift-dns/dns-default-mctfq"
Oct 30 03:58:01.843219 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843013    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-sysctl-d\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.843219 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843019    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade9bd13-6259-48aa-92b2-52f875b960e8-tmp-dir\") pod \"node-resolver-8g477\" (UID: \"ade9bd13-6259-48aa-92b2-52f875b960e8\") " pod="openshift-dns/node-resolver-8g477"
Oct 30 03:58:01.843219 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843035    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-systemd\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.843219 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843086    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-lib-modules\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.843219 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843113    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64f1a482-331f-4490-b03f-f41facd80cfe-tmp\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.843219 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843138    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-run-netns\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.843219 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843162    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-cni-bin\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.843219 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843186    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qr8r\" (UniqueName: \"kubernetes.io/projected/4dec34b4-fb69-459e-b130-ca1716d23765-kube-api-access-4qr8r\") pod \"network-metrics-daemon-dpdr4\" (UID: \"4dec34b4-fb69-459e-b130-ca1716d23765\") " pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:58:01.843219 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843189    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64f1a482-331f-4490-b03f-f41facd80cfe-tmp\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.843219 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843217    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-run-systemd\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.843219 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843223    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-cni-binary-copy\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843288    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-run-ovn\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843322    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-var-lib-kubelet\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843350    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grbsb\" (UniqueName: \"kubernetes.io/projected/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-kube-api-access-grbsb\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843420    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30e672ce-27b7-4997-a887-b834e6a6797d-metrics-client-ca\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843484    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843503    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a6755b1f-578f-4e75-86df-be3db7533610-config-volume\") pod \"dns-default-mctfq\" (UID: \"a6755b1f-578f-4e75-86df-be3db7533610\") " pod="openshift-dns/dns-default-mctfq"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843524    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/455076cd-b484-44b6-8a1b-b4c3c0dccab8-serviceca\") pod \"node-ca-lrcg5\" (UID: \"455076cd-b484-44b6-8a1b-b4c3c0dccab8\") " pod="openshift-image-registry/node-ca-lrcg5"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843554    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-modprobe-d\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843578    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-multus-socket-dir-parent\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843601    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6755b1f-578f-4e75-86df-be3db7533610-metrics-tls\") pod \"dns-default-mctfq\" (UID: \"a6755b1f-578f-4e75-86df-be3db7533610\") " pod="openshift-dns/dns-default-mctfq"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843653    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-sys\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843673    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30e672ce-27b7-4997-a887-b834e6a6797d-metrics-client-ca\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843673    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843699    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a6755b1f-578f-4e75-86df-be3db7533610-config-volume\") pod \"dns-default-mctfq\" (UID: \"a6755b1f-578f-4e75-86df-be3db7533610\") " pod="openshift-dns/dns-default-mctfq"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843685    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/64f1a482-331f-4490-b03f-f41facd80cfe-etc-tuned\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843747    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843749    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/64f1a482-331f-4490-b03f-f41facd80cfe-etc-tuned\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843790    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/97de5e90-d514-4b8c-8917-2e9894978aaa-env-overrides\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843793    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/455076cd-b484-44b6-8a1b-b4c3c0dccab8-serviceca\") pod \"node-ca-lrcg5\" (UID: \"455076cd-b484-44b6-8a1b-b4c3c0dccab8\") " pod="openshift-image-registry/node-ca-lrcg5"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843809    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bad69b26-9de8-4c2b-b19c-8a96271f5c86-original-pull-secret\") pod \"global-pull-secret-syncer-cbfm8\" (UID: \"bad69b26-9de8-4c2b-b19c-8a96271f5c86\") " pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843819    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6755b1f-578f-4e75-86df-be3db7533610-metrics-tls\") pod \"dns-default-mctfq\" (UID: \"a6755b1f-578f-4e75-86df-be3db7533610\") " pod="openshift-dns/dns-default-mctfq"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843825    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843855    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843887    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-socket-dir\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843925    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rszkc\" (UniqueName: \"kubernetes.io/projected/64f1a482-331f-4490-b03f-f41facd80cfe-kube-api-access-rszkc\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843961    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/97de5e90-d514-4b8c-8917-2e9894978aaa-env-overrides\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843971    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-cnibin\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.843998    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30e672ce-27b7-4997-a887-b834e6a6797d-sys\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844025    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-multus-daemon-config\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844067    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bad69b26-9de8-4c2b-b19c-8a96271f5c86-original-pull-secret\") pod \"global-pull-secret-syncer-cbfm8\" (UID: \"bad69b26-9de8-4c2b-b19c-8a96271f5c86\") " pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:58:01.844070 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844080    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844080    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844130    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-run-multus-certs\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844150    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-cnibin\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844192    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844263    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h268\" (UniqueName: \"kubernetes.io/projected/bee7738d-afa2-431d-ba81-2d14f142902c-kube-api-access-6h268\") pod \"ingress-canary-2ldpw\" (UID: \"bee7738d-afa2-431d-ba81-2d14f142902c\") " pod="openshift-ingress-canary/ingress-canary-2ldpw"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844320    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-multus-daemon-config\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844332    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/67ea77e7-74cc-4bc6-ae16-5a5a797f5097-iptables-alerter-script\") pod \"iptables-alerter-ttsr6\" (UID: \"67ea77e7-74cc-4bc6-ae16-5a5a797f5097\") " pod="openshift-network-operator/iptables-alerter-ttsr6"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844389    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844475    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-kubelet\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844506    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-accelerators-collector-config\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844513    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/67ea77e7-74cc-4bc6-ae16-5a5a797f5097-iptables-alerter-script\") pod \"iptables-alerter-ttsr6\" (UID: \"67ea77e7-74cc-4bc6-ae16-5a5a797f5097\") " pod="openshift-network-operator/iptables-alerter-ttsr6"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844525    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-data-volume\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844555    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844572    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6msml\" (UniqueName: \"kubernetes.io/projected/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-kube-api-access-6msml\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844588    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67ea77e7-74cc-4bc6-ae16-5a5a797f5097-host-slash\") pod \"iptables-alerter-ttsr6\" (UID: \"67ea77e7-74cc-4bc6-ae16-5a5a797f5097\") " pod="openshift-network-operator/iptables-alerter-ttsr6"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844593    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-data-volume\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844602    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-etc-selinux\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844618    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/97de5e90-d514-4b8c-8917-2e9894978aaa-ovn-node-metrics-cert\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844697    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-accelerators-collector-config\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844730    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-multus-conf-dir\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844775    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bee7738d-afa2-431d-ba81-2d14f142902c-cert\") pod \"ingress-canary-2ldpw\" (UID: \"bee7738d-afa2-431d-ba81-2d14f142902c\") " pod="openshift-ingress-canary/ingress-canary-2ldpw"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844804    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844855    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/97de5e90-d514-4b8c-8917-2e9894978aaa-ovn-node-metrics-cert\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844889    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e385d23f-6371-4f69-97b1-6a10303c048f-agent-certs\") pod \"konnectivity-agent-vnv9w\" (UID: \"e385d23f-6371-4f69-97b1-6a10303c048f\") " pod="kube-system/konnectivity-agent-vnv9w"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844920    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-sysconfig\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844950    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/97de5e90-d514-4b8c-8917-2e9894978aaa-ovnkube-config\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844986    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bee7738d-afa2-431d-ba81-2d14f142902c-cert\") pod \"ingress-canary-2ldpw\" (UID: \"bee7738d-afa2-431d-ba81-2d14f142902c\") " pod="openshift-ingress-canary/ingress-canary-2ldpw"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.844991    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z24jg\" (UniqueName: \"kubernetes.io/projected/97de5e90-d514-4b8c-8917-2e9894978aaa-kube-api-access-z24jg\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.845126    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/97de5e90-d514-4b8c-8917-2e9894978aaa-ovnkube-config\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.845489 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.845137    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e385d23f-6371-4f69-97b1-6a10303c048f-agent-certs\") pod \"konnectivity-agent-vnv9w\" (UID: \"e385d23f-6371-4f69-97b1-6a10303c048f\") " pod="kube-system/konnectivity-agent-vnv9w"
Oct 30 03:58:01.861762 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.861738    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfdvg\" (UniqueName: \"kubernetes.io/projected/67ea77e7-74cc-4bc6-ae16-5a5a797f5097-kube-api-access-zfdvg\") pod \"iptables-alerter-ttsr6\" (UID: \"67ea77e7-74cc-4bc6-ae16-5a5a797f5097\") " pod="openshift-network-operator/iptables-alerter-ttsr6"
Oct 30 03:58:01.861908 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.861887    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg642\" (UniqueName: \"kubernetes.io/projected/455076cd-b484-44b6-8a1b-b4c3c0dccab8-kube-api-access-zg642\") pod \"node-ca-lrcg5\" (UID: \"455076cd-b484-44b6-8a1b-b4c3c0dccab8\") " pod="openshift-image-registry/node-ca-lrcg5"
Oct 30 03:58:01.862016 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.862001    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxh5g\" (UniqueName: \"kubernetes.io/projected/baf40c59-c04b-4c27-b080-6c55887c5c4c-kube-api-access-bxh5g\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:58:01.862065 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.862054    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwtgk\" (UniqueName: \"kubernetes.io/projected/d5e46634-9969-4d78-bb91-e598e950b1b0-kube-api-access-jwtgk\") pod \"network-check-target-9mktl\" (UID: \"d5e46634-9969-4d78-bb91-e598e950b1b0\") " pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:58:01.862373 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.862357    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kq65\" (UniqueName: \"kubernetes.io/projected/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-kube-api-access-6kq65\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:58:01.863025 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.863000    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvzdn\" (UniqueName: \"kubernetes.io/projected/ade9bd13-6259-48aa-92b2-52f875b960e8-kube-api-access-lvzdn\") pod \"node-resolver-8g477\" (UID: \"ade9bd13-6259-48aa-92b2-52f875b960e8\") " pod="openshift-dns/node-resolver-8g477"
Oct 30 03:58:01.863114 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.863101    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt7mf\" (UniqueName: \"kubernetes.io/projected/a6755b1f-578f-4e75-86df-be3db7533610-kube-api-access-dt7mf\") pod \"dns-default-mctfq\" (UID: \"a6755b1f-578f-4e75-86df-be3db7533610\") " pod="openshift-dns/dns-default-mctfq"
Oct 30 03:58:01.863471 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.863451    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hl75\" (UniqueName: \"kubernetes.io/projected/30e672ce-27b7-4997-a887-b834e6a6797d-kube-api-access-5hl75\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:58:01.864784 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.864766    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qr8r\" (UniqueName: \"kubernetes.io/projected/4dec34b4-fb69-459e-b130-ca1716d23765-kube-api-access-4qr8r\") pod \"network-metrics-daemon-dpdr4\" (UID: \"4dec34b4-fb69-459e-b130-ca1716d23765\") " pod="openshift-multus/network-metrics-daemon-dpdr4"
Oct 30 03:58:01.864952 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.864936    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rszkc\" (UniqueName: \"kubernetes.io/projected/64f1a482-331f-4490-b03f-f41facd80cfe-kube-api-access-rszkc\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.865027 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.864978    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h268\" (UniqueName: \"kubernetes.io/projected/bee7738d-afa2-431d-ba81-2d14f142902c-kube-api-access-6h268\") pod \"ingress-canary-2ldpw\" (UID: \"bee7738d-afa2-431d-ba81-2d14f142902c\") " pod="openshift-ingress-canary/ingress-canary-2ldpw"
Oct 30 03:58:01.865408 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.865393    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grbsb\" (UniqueName: \"kubernetes.io/projected/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-kube-api-access-grbsb\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.866737 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.866721    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6msml\" (UniqueName: \"kubernetes.io/projected/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-kube-api-access-6msml\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:58:01.866935 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.866917    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z24jg\" (UniqueName: \"kubernetes.io/projected/97de5e90-d514-4b8c-8917-2e9894978aaa-kube-api-access-z24jg\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.945424 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945356    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-cnibin\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:58:01.945424 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945393    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-kubelet\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.945424 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945407    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67ea77e7-74cc-4bc6-ae16-5a5a797f5097-host-slash\") pod \"iptables-alerter-ttsr6\" (UID: \"67ea77e7-74cc-4bc6-ae16-5a5a797f5097\") " pod="openshift-network-operator/iptables-alerter-ttsr6"
Oct 30 03:58:01.945424 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945422    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-etc-selinux\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:58:01.945664 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945467    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-cnibin\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:58:01.945664 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945493    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-multus-conf-dir\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.945664 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945509    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-sysconfig\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.945664 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945509    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67ea77e7-74cc-4bc6-ae16-5a5a797f5097-host-slash\") pod \"iptables-alerter-ttsr6\" (UID: \"67ea77e7-74cc-4bc6-ae16-5a5a797f5097\") " pod="openshift-network-operator/iptables-alerter-ttsr6"
Oct 30 03:58:01.945664 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945524    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-hostroot\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.945664 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945532    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-etc-selinux\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:58:01.945664 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945539    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:58:01.945664 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945557    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-multus-conf-dir\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.945664 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945563    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-var-lib-kubelet\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.945664 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945580    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-systemd-units\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.945664 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945602    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.945664 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945621    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-sysconfig\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.945664 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945633    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-systemd-units\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.945664 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945638    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-var-lib-cni-multus\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.945664 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945580    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:58:01.945664 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945664    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945673    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-kubelet\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945667    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30e672ce-27b7-4997-a887-b834e6a6797d-root\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945575    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-hostroot\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945702    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-var-lib-cni-multus\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945711    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-var-lib-kubelet\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945745    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30e672ce-27b7-4997-a887-b834e6a6797d-root\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945790    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-run\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945813    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-run\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945813    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-host\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945837    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-multus-cni-dir\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945849    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-host\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945863    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bad69b26-9de8-4c2b-b19c-8a96271f5c86-kubelet-config\") pod \"global-pull-secret-syncer-cbfm8\" (UID: \"bad69b26-9de8-4c2b-b19c-8a96271f5c86\") " pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945881    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-system-cni-dir\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945896    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-run-netns\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945911    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bad69b26-9de8-4c2b-b19c-8a96271f5c86-dbus\") pod \"global-pull-secret-syncer-cbfm8\" (UID: \"bad69b26-9de8-4c2b-b19c-8a96271f5c86\") " pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945931    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bad69b26-9de8-4c2b-b19c-8a96271f5c86-kubelet-config\") pod \"global-pull-secret-syncer-cbfm8\" (UID: \"bad69b26-9de8-4c2b-b19c-8a96271f5c86\") " pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945934    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-registration-dir\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945969    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-system-cni-dir\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945973    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-multus-cni-dir\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945985    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-registration-dir\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945986    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bad69b26-9de8-4c2b-b19c-8a96271f5c86-dbus\") pod \"global-pull-secret-syncer-cbfm8\" (UID: \"bad69b26-9de8-4c2b-b19c-8a96271f5c86\") " pod="kube-system/global-pull-secret-syncer-cbfm8"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945986    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-run-netns\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.945970    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-slash\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946006    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-slash\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946045    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-var-lib-openvswitch\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946083    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-run-openvswitch\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946107    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-log-socket\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946141    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-var-lib-openvswitch\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946154    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-run-openvswitch\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946153    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-log-socket\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946209    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-wtmp\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946252    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-system-cni-dir\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946269    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-crio-socket\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946287    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-sysctl-conf\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946309    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-var-lib-cni-bin\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946323    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8-system-cni-dir\") pod \"multus-additional-cni-plugins-9nxvt\" (UID: \"4ef4cdba-d13e-4e6b-a4e6-d722d58b55e8\") " pod="openshift-multus/multus-additional-cni-plugins-9nxvt"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946332    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30e672ce-27b7-4997-a887-b834e6a6797d-node-exporter-wtmp\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946346    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/95dfe3d1-42fa-41b5-a6d3-4774214d54b1-crio-socket\") pod \"insights-runtime-extractor-8qhbh\" (UID: \"95dfe3d1-42fa-41b5-a6d3-4774214d54b1\") " pod="openshift-insights/insights-runtime-extractor-8qhbh"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946348    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-var-lib-cni-bin\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946360    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-etc-kubernetes\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946388    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-sysctl-conf\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.946365 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946390    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ade9bd13-6259-48aa-92b2-52f875b960e8-hosts-file\") pod \"node-resolver-8g477\" (UID: \"ade9bd13-6259-48aa-92b2-52f875b960e8\") " pod="openshift-dns/node-resolver-8g477"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946424    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-device-dir\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946443    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-etc-kubernetes\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946448    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-sys-fs\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946459    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ade9bd13-6259-48aa-92b2-52f875b960e8-hosts-file\") pod \"node-resolver-8g477\" (UID: \"ade9bd13-6259-48aa-92b2-52f875b960e8\") " pod="openshift-dns/node-resolver-8g477"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946479    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-kubernetes\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946492    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-sys-fs\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946506    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-device-dir\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946509    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-node-log\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946541    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-kubernetes\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946545    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-node-log\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946541    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-run-k8s-cni-cncf-io\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946571    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-run-k8s-cni-cncf-io\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946575    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/455076cd-b484-44b6-8a1b-b4c3c0dccab8-host\") pod \"node-ca-lrcg5\" (UID: \"455076cd-b484-44b6-8a1b-b4c3c0dccab8\") " pod="openshift-image-registry/node-ca-lrcg5"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946600    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/455076cd-b484-44b6-8a1b-b4c3c0dccab8-host\") pod \"node-ca-lrcg5\" (UID: \"455076cd-b484-44b6-8a1b-b4c3c0dccab8\") " pod="openshift-image-registry/node-ca-lrcg5"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946603    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946627    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-etc-openvswitch\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946650    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-cni-netd\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946667    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946679    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-sysctl-d\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946704    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-systemd\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946713    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-etc-openvswitch\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946726    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-lib-modules\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946729    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-cni-netd\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946753    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-run-netns\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946764    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-systemd\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946768    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-cni-bin\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946789    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-run-systemd\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946793    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-cni-bin\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946806    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-run-ovn\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946817    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-lib-modules\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946816    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-run-systemd\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946829    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-run-netns\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946830    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-var-lib-kubelet\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946849    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-var-lib-kubelet\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946850    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-run-ovn\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946868    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-modprobe-d\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946820    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-sysctl-d\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946889    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-multus-socket-dir-parent\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946909    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-sys\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946924    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946939    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-socket-dir\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946953    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-cnibin\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946953    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-etc-modprobe-d\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946966    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30e672ce-27b7-4997-a887-b834e6a6797d-sys\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.946993    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97de5e90-d514-4b8c-8917-2e9894978aaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-244hd\" (UID: \"97de5e90-d514-4b8c-8917-2e9894978aaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.947010    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64f1a482-331f-4490-b03f-f41facd80cfe-sys\") pod \"tuned-b88ss\" (UID: \"64f1a482-331f-4490-b03f-f41facd80cfe\") " pod="openshift-cluster-node-tuning-operator/tuned-b88ss"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.947007    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-multus-socket-dir-parent\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.947023    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-cnibin\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.947030    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-run-multus-certs\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.947038    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30e672ce-27b7-4997-a887-b834e6a6797d-sys\") pod \"node-exporter-dgvb4\" (UID: \"30e672ce-27b7-4997-a887-b834e6a6797d\") " pod="openshift-monitoring/node-exporter-dgvb4"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.947032    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/baf40c59-c04b-4c27-b080-6c55887c5c4c-socket-dir\") pod \"aws-ebs-csi-driver-node-7gkv4\" (UID: \"baf40c59-c04b-4c27-b080-6c55887c5c4c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gkv4"
Oct 30 03:58:01.947675 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:01.947089    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c9f3a00-e77c-4946-b5bb-3a0cb171b83e-host-run-multus-certs\") pod \"multus-zl7qk\" (UID: \"3c9f3a00-e77c-4946-b5bb-3a0cb171b83e\") " pod="openshift-multus/multus-zl7qk"
Oct 30 03:58:04.043823 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:04.043792    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:04.465927 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:04.465851    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:04.775256 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:04.775216    4874 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 03:58:04.879827 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:04.879804    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mctfq"
Oct 30 03:58:04.881382 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:04.881362    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mctfq"
Oct 30 03:58:06.390293 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:06.390226    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:58:07.811671 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:07.811630    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-244hd"
Oct 30 03:58:11.674441 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.674412    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-84c6c88b65-z5b2c"]
Oct 30 03:58:11.676293 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.676277    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:11.691989 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.691969    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 03:58:11.703545 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.703500    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 03:58:11.703545 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.703545    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 03:58:11.703683 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.703551    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 03:58:11.703683 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.703615    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-gmjld\""
Oct 30 03:58:11.703683 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.703665    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 03:58:11.703683 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.703666    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 03:58:11.727569 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.727552    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 03:58:11.727652 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.727571    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 03:58:11.728145 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.728128    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84c6c88b65-z5b2c"]
Oct 30 03:58:11.796786 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.796758    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff-oauth-serving-cert\") pod \"console-84c6c88b65-z5b2c\" (UID: \"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff\") " pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:11.796922 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.796791    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff-service-ca\") pod \"console-84c6c88b65-z5b2c\" (UID: \"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff\") " pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:11.796922 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.796822    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff-console-serving-cert\") pod \"console-84c6c88b65-z5b2c\" (UID: \"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff\") " pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:11.797036 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.796925    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff-console-config\") pod \"console-84c6c88b65-z5b2c\" (UID: \"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff\") " pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:11.797036 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.797004    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff-trusted-ca-bundle\") pod \"console-84c6c88b65-z5b2c\" (UID: \"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff\") " pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:11.797139 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.797044    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff-console-oauth-config\") pod \"console-84c6c88b65-z5b2c\" (UID: \"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff\") " pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:11.797139 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.797072    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff-tmp\") pod \"console-84c6c88b65-z5b2c\" (UID: \"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff\") " pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:11.797261 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.797189    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blpbz\" (UniqueName: \"kubernetes.io/projected/b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff-kube-api-access-blpbz\") pod \"console-84c6c88b65-z5b2c\" (UID: \"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff\") " pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:11.897844 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.897812    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff-service-ca\") pod \"console-84c6c88b65-z5b2c\" (UID: \"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff\") " pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:11.897950 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.897860    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff-console-serving-cert\") pod \"console-84c6c88b65-z5b2c\" (UID: \"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff\") " pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:11.898040 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.898024    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff-console-config\") pod \"console-84c6c88b65-z5b2c\" (UID: \"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff\") " pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:11.898075 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.898053    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff-trusted-ca-bundle\") pod \"console-84c6c88b65-z5b2c\" (UID: \"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff\") " pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:11.898075 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.898072    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff-console-oauth-config\") pod \"console-84c6c88b65-z5b2c\" (UID: \"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff\") " pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:11.898163 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.898088    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff-tmp\") pod \"console-84c6c88b65-z5b2c\" (UID: \"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff\") " pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:11.898163 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.898126    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blpbz\" (UniqueName: \"kubernetes.io/projected/b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff-kube-api-access-blpbz\") pod \"console-84c6c88b65-z5b2c\" (UID: \"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff\") " pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:11.898163 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.898156    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff-oauth-serving-cert\") pod \"console-84c6c88b65-z5b2c\" (UID: \"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff\") " pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:11.898338 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.898310    4874 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 03:58:11.898601 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.898576    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff-tmp\") pod \"console-84c6c88b65-z5b2c\" (UID: \"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff\") " pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:11.898791 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.898759    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff-console-config\") pod \"console-84c6c88b65-z5b2c\" (UID: \"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff\") " pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:11.898941 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.898923    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff-trusted-ca-bundle\") pod \"console-84c6c88b65-z5b2c\" (UID: \"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff\") " pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:11.899113 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.899089    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff-oauth-serving-cert\") pod \"console-84c6c88b65-z5b2c\" (UID: \"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff\") " pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:11.899184 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.899095    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff-service-ca\") pod \"console-84c6c88b65-z5b2c\" (UID: \"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff\") " pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:11.900997 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.900977    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff-console-serving-cert\") pod \"console-84c6c88b65-z5b2c\" (UID: \"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff\") " pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:11.900997 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.900986    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff-console-oauth-config\") pod \"console-84c6c88b65-z5b2c\" (UID: \"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff\") " pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:11.920799 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.920777    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blpbz\" (UniqueName: \"kubernetes.io/projected/b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff-kube-api-access-blpbz\") pod \"console-84c6c88b65-z5b2c\" (UID: \"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff\") " pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:11.985555 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:11.985523    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:12.277413 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:12.277387    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84c6c88b65-z5b2c"]
Oct 30 03:58:12.278838 ip-10-0-138-160 kubenswrapper[4874]: W1030 03:58:12.278805    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb188bb6b_c5d1_4b71_8b5e_0f1aadc20aff.slice/crio-b512605002ccbd29dfdac77361173346629a16ddea867d17ebd0260b67fc844b WatchSource:0}: Error finding container b512605002ccbd29dfdac77361173346629a16ddea867d17ebd0260b67fc844b: Status 404 returned error can't find the container with id b512605002ccbd29dfdac77361173346629a16ddea867d17ebd0260b67fc844b
Oct 30 03:58:12.280843 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:12.280824    4874 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:58:12.801251 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:12.801205    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c6c88b65-z5b2c" event={"ID":"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff","Type":"ContainerStarted","Data":"b512605002ccbd29dfdac77361173346629a16ddea867d17ebd0260b67fc844b"}
Oct 30 03:58:14.751078 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:14.751055    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vnv9w"
Oct 30 03:58:14.808684 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:14.808650    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c6c88b65-z5b2c" event={"ID":"b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff","Type":"ContainerStarted","Data":"f4fdf040796a109a7f34863404fb939d6db897bed01b0432ebb6ebf92ed97492"}
Oct 30 03:58:14.849788 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:14.849733    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-84c6c88b65-z5b2c" podStartSLOduration=1.480699934 podStartE2EDuration="3.849721253s" totalImagesPullingTime="2.369021319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:58:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:58:14.84968714 +0000 UTC m=+14.262082712" watchObservedRunningTime="2026-10-30 03:58:14.849721253 +0000 UTC m=+14.262116826"
Oct 30 03:58:16.321776 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.321742    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 03:58:16.324163 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.324146    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.363063 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.363036    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 03:58:16.363655 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.363636    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 03:58:16.419067 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.419039    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 03:58:16.419067 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.419060    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 03:58:16.419259 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.419054    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 03:58:16.419259 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.419077    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 03:58:16.419259 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.419088    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 03:58:16.419259 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.419094    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 03:58:16.419259 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.419040    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 03:58:16.419259 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.419141    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pttks\""
Oct 30 03:58:16.419259 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.419167    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 03:58:16.431541 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.431522    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c679faf6-9b18-4a4b-9dee-d5946c10f168-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.431594 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.431549    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c679faf6-9b18-4a4b-9dee-d5946c10f168-web-config\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.431594 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.431572    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c679faf6-9b18-4a4b-9dee-d5946c10f168-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.431594 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.431587    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c679faf6-9b18-4a4b-9dee-d5946c10f168-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.431712 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.431603    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c679faf6-9b18-4a4b-9dee-d5946c10f168-config-volume\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.431712 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.431654    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c679faf6-9b18-4a4b-9dee-d5946c10f168-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.431846 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.431709    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c679faf6-9b18-4a4b-9dee-d5946c10f168-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.431846 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.431729    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkv9l\" (UniqueName: \"kubernetes.io/projected/c679faf6-9b18-4a4b-9dee-d5946c10f168-kube-api-access-bkv9l\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.431952 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.431867    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c679faf6-9b18-4a4b-9dee-d5946c10f168-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.431952 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.431907    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c679faf6-9b18-4a4b-9dee-d5946c10f168-config-out\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.432018 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.431995    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c679faf6-9b18-4a4b-9dee-d5946c10f168-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.432052 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.432017    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c679faf6-9b18-4a4b-9dee-d5946c10f168-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.432052 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.432040    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c679faf6-9b18-4a4b-9dee-d5946c10f168-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.532856 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.532816    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c679faf6-9b18-4a4b-9dee-d5946c10f168-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.532856 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.532864    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c679faf6-9b18-4a4b-9dee-d5946c10f168-config-out\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.532998 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.532891    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c679faf6-9b18-4a4b-9dee-d5946c10f168-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.532998 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.532908    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c679faf6-9b18-4a4b-9dee-d5946c10f168-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.532998 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.532926    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c679faf6-9b18-4a4b-9dee-d5946c10f168-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.532998 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.532946    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c679faf6-9b18-4a4b-9dee-d5946c10f168-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.532998 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.532962    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c679faf6-9b18-4a4b-9dee-d5946c10f168-web-config\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.532998 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.532979    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c679faf6-9b18-4a4b-9dee-d5946c10f168-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.532998 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.532995    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c679faf6-9b18-4a4b-9dee-d5946c10f168-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.533333 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.533019    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c679faf6-9b18-4a4b-9dee-d5946c10f168-config-volume\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.533333 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.533041    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c679faf6-9b18-4a4b-9dee-d5946c10f168-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.533333 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.533068    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c679faf6-9b18-4a4b-9dee-d5946c10f168-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.533333 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.533091    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkv9l\" (UniqueName: \"kubernetes.io/projected/c679faf6-9b18-4a4b-9dee-d5946c10f168-kube-api-access-bkv9l\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.533707 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.533680    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c679faf6-9b18-4a4b-9dee-d5946c10f168-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.534919 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.534895    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c679faf6-9b18-4a4b-9dee-d5946c10f168-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.535332 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.535311    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c679faf6-9b18-4a4b-9dee-d5946c10f168-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.536197 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.536167    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c679faf6-9b18-4a4b-9dee-d5946c10f168-web-config\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.536296 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.536255    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c679faf6-9b18-4a4b-9dee-d5946c10f168-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.536427 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.536406    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c679faf6-9b18-4a4b-9dee-d5946c10f168-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.536463 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.536441    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c679faf6-9b18-4a4b-9dee-d5946c10f168-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.536500 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.536462    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c679faf6-9b18-4a4b-9dee-d5946c10f168-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.536973 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.536948    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c679faf6-9b18-4a4b-9dee-d5946c10f168-config-out\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.537053 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.536995    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c679faf6-9b18-4a4b-9dee-d5946c10f168-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.537703 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.537685    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c679faf6-9b18-4a4b-9dee-d5946c10f168-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.537752 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.537700    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c679faf6-9b18-4a4b-9dee-d5946c10f168-config-volume\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.555305 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.555286    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkv9l\" (UniqueName: \"kubernetes.io/projected/c679faf6-9b18-4a4b-9dee-d5946c10f168-kube-api-access-bkv9l\") pod \"alertmanager-main-0\" (UID: \"c679faf6-9b18-4a4b-9dee-d5946c10f168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.640757 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.640708    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 03:58:16.967434 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:16.967400    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 03:58:17.819304 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:17.819272    4874 generic.go:356] "Generic (PLEG): container finished" podID="c679faf6-9b18-4a4b-9dee-d5946c10f168" containerID="bad18ff71ae6a365c4eb216ca91cc2f7e8fae27e3556dbf6f5496c5365d9784b" exitCode=0
Oct 30 03:58:17.819588 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:17.819318    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c679faf6-9b18-4a4b-9dee-d5946c10f168","Type":"ContainerDied","Data":"bad18ff71ae6a365c4eb216ca91cc2f7e8fae27e3556dbf6f5496c5365d9784b"}
Oct 30 03:58:17.819588 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:17.819395    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c679faf6-9b18-4a4b-9dee-d5946c10f168","Type":"ContainerStarted","Data":"0f41256ee39207f58d4da733b19cd1d826b9a5eddafb5dc76faa6e632f3a5908"}
Oct 30 03:58:19.829825 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:19.829792    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c679faf6-9b18-4a4b-9dee-d5946c10f168","Type":"ContainerStarted","Data":"8590e0a605260f02d14b721a85a8cb192781b354602a23b6a62313aefbc700d3"}
Oct 30 03:58:19.830201 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:19.829872    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c679faf6-9b18-4a4b-9dee-d5946c10f168","Type":"ContainerStarted","Data":"5013a2799e9c056d905f7315c1fb6262a1b92218a56e97fee6e5d323eeb6195a"}
Oct 30 03:58:19.830201 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:19.829922    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c679faf6-9b18-4a4b-9dee-d5946c10f168","Type":"ContainerStarted","Data":"78cd6d4acef5a203464474a582dd2704065516810316f77d32692da18244970e"}
Oct 30 03:58:19.830201 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:19.829954    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c679faf6-9b18-4a4b-9dee-d5946c10f168","Type":"ContainerStarted","Data":"ecf2d08b5575c39fad03f7e78d0a94d2fc04b0f253010ecd7b57e81c8300db12"}
Oct 30 03:58:19.830201 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:19.829978    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c679faf6-9b18-4a4b-9dee-d5946c10f168","Type":"ContainerStarted","Data":"74cfe9d3774991a2b5321128cfec1f260b4d2d2fa893cb316e0d486d08bc83bc"}
Oct 30 03:58:20.835204 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:20.835171    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c679faf6-9b18-4a4b-9dee-d5946c10f168","Type":"ContainerStarted","Data":"d980caba9fc3ab1a0b162a3ea5f79b9c74b305e96efced72de8e0204bb4aea17"}
Oct 30 03:58:20.920740 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:20.920677    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.152857447 podStartE2EDuration="4.92065869s" totalImagesPullingTime="2.755402659s" totalInitContainerRuntime="12.398584ms" isStatefulPod=false podCreationTimestamp="2026-10-30 03:58:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:58:20.908413122 +0000 UTC m=+20.320808694" watchObservedRunningTime="2026-10-30 03:58:20.92065869 +0000 UTC m=+20.333054264"
Oct 30 03:58:21.985976 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:21.985941    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:21.986426 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:21.986007    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:21.990916 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:21.990893    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:22.845926 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:22.845895    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84c6c88b65-z5b2c"
Oct 30 03:58:25.607921 ip-10-0-138-160 kubenswrapper[4874]: E1030 03:58:25.607886    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 03:58:36.392466 ip-10-0-138-160 kubenswrapper[4874]: I1030 03:58:36.392433    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9mktl"
Oct 30 03:58:55.607647 ip-10-0-138-160 kubenswrapper[4874]: E1030 03:58:55.607610    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 03:59:25.610788 ip-10-0-138-160 kubenswrapper[4874]: E1030 03:59:25.610745    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 03:59:55.607689 ip-10-0-138-160 kubenswrapper[4874]: E1030 03:59:55.607654    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:00:25.607492 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:00:25.607417    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:00:40.762873 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:40.762843    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn"]
Oct 30 04:00:40.765931 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:40.765916    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn"
Oct 30 04:00:40.794878 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:40.794854    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 04:00:40.795102 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:40.795088    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-rhdgb\""
Oct 30 04:00:40.795145 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:40.795094    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 04:00:40.824167 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:40.824144    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn"]
Oct 30 04:00:40.844484 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:40.844455    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73d043bc-6a33-439f-b75e-2716b30bce34-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn\" (UID: \"73d043bc-6a33-439f-b75e-2716b30bce34\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn"
Oct 30 04:00:40.844592 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:40.844488    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73d043bc-6a33-439f-b75e-2716b30bce34-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn\" (UID: \"73d043bc-6a33-439f-b75e-2716b30bce34\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn"
Oct 30 04:00:40.844661 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:40.844642    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trvnp\" (UniqueName: \"kubernetes.io/projected/73d043bc-6a33-439f-b75e-2716b30bce34-kube-api-access-trvnp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn\" (UID: \"73d043bc-6a33-439f-b75e-2716b30bce34\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn"
Oct 30 04:00:40.887651 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:40.887623    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7"]
Oct 30 04:00:40.890835 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:40.890820    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7"
Oct 30 04:00:40.905397 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:40.905379    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-bnbhg\""
Oct 30 04:00:40.909833 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:40.909801    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7"]
Oct 30 04:00:40.945608 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:40.945582    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aeafd1bc-0c7a-4268-a363-7c12399045b3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7\" (UID: \"aeafd1bc-0c7a-4268-a363-7c12399045b3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7"
Oct 30 04:00:40.945703 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:40.945636    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73d043bc-6a33-439f-b75e-2716b30bce34-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn\" (UID: \"73d043bc-6a33-439f-b75e-2716b30bce34\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn"
Oct 30 04:00:40.945751 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:40.945696    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73d043bc-6a33-439f-b75e-2716b30bce34-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn\" (UID: \"73d043bc-6a33-439f-b75e-2716b30bce34\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn"
Oct 30 04:00:40.945793 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:40.945763    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trvnp\" (UniqueName: \"kubernetes.io/projected/73d043bc-6a33-439f-b75e-2716b30bce34-kube-api-access-trvnp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn\" (UID: \"73d043bc-6a33-439f-b75e-2716b30bce34\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn"
Oct 30 04:00:40.945852 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:40.945795    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aeafd1bc-0c7a-4268-a363-7c12399045b3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7\" (UID: \"aeafd1bc-0c7a-4268-a363-7c12399045b3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7"
Oct 30 04:00:40.945852 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:40.945828    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k9n8\" (UniqueName: \"kubernetes.io/projected/aeafd1bc-0c7a-4268-a363-7c12399045b3-kube-api-access-2k9n8\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7\" (UID: \"aeafd1bc-0c7a-4268-a363-7c12399045b3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7"
Oct 30 04:00:40.946040 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:40.946024    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73d043bc-6a33-439f-b75e-2716b30bce34-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn\" (UID: \"73d043bc-6a33-439f-b75e-2716b30bce34\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn"
Oct 30 04:00:40.946120 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:40.946063    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73d043bc-6a33-439f-b75e-2716b30bce34-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn\" (UID: \"73d043bc-6a33-439f-b75e-2716b30bce34\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn"
Oct 30 04:00:40.968392 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:40.968368    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trvnp\" (UniqueName: \"kubernetes.io/projected/73d043bc-6a33-439f-b75e-2716b30bce34-kube-api-access-trvnp\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn\" (UID: \"73d043bc-6a33-439f-b75e-2716b30bce34\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn"
Oct 30 04:00:41.046782 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.046725    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aeafd1bc-0c7a-4268-a363-7c12399045b3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7\" (UID: \"aeafd1bc-0c7a-4268-a363-7c12399045b3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7"
Oct 30 04:00:41.046871 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.046784    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aeafd1bc-0c7a-4268-a363-7c12399045b3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7\" (UID: \"aeafd1bc-0c7a-4268-a363-7c12399045b3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7"
Oct 30 04:00:41.046871 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.046817    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k9n8\" (UniqueName: \"kubernetes.io/projected/aeafd1bc-0c7a-4268-a363-7c12399045b3-kube-api-access-2k9n8\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7\" (UID: \"aeafd1bc-0c7a-4268-a363-7c12399045b3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7"
Oct 30 04:00:41.047151 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.047134    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aeafd1bc-0c7a-4268-a363-7c12399045b3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7\" (UID: \"aeafd1bc-0c7a-4268-a363-7c12399045b3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7"
Oct 30 04:00:41.047191 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.047150    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aeafd1bc-0c7a-4268-a363-7c12399045b3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7\" (UID: \"aeafd1bc-0c7a-4268-a363-7c12399045b3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7"
Oct 30 04:00:41.069681 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.069657    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k9n8\" (UniqueName: \"kubernetes.io/projected/aeafd1bc-0c7a-4268-a363-7c12399045b3-kube-api-access-2k9n8\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7\" (UID: \"aeafd1bc-0c7a-4268-a363-7c12399045b3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7"
Oct 30 04:00:41.081397 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.081377    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn"
Oct 30 04:00:41.129085 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.129059    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"]
Oct 30 04:00:41.138878 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.138844    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:00:41.164594 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.164574    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2vrdc\""
Oct 30 04:00:41.166523 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.166499    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"]
Oct 30 04:00:41.167445 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.167426    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 04:00:41.169471 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.169451    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 04:00:41.169573 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.169511    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 04:00:41.169573 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.169531    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 04:00:41.204971 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.204948    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7"
Oct 30 04:00:41.247839 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.247703    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-wg2xp\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:00:41.247956 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.247847    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-wg2xp\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:00:41.247956 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.247898    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-wg2xp\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:00:41.247956 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.247931    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-wg2xp\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:00:41.248062 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.248044    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-wg2xp\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:00:41.248104 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.248073    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-wg2xp\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:00:41.248104 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.248097    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ztvw\" (UniqueName: \"kubernetes.io/projected/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-kube-api-access-4ztvw\") pod \"istiod-openshift-gateway-7876bfcb74-wg2xp\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:00:41.274845 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.274813    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 04:00:41.274932 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.274894    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 04:00:41.348969 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.348922    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-wg2xp\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:00:41.348969 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.348962    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-wg2xp\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:00:41.349133 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.348982    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-wg2xp\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:00:41.349199 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.349125    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-wg2xp\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:00:41.349199 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.349166    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-wg2xp\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:00:41.349319 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.349197    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ztvw\" (UniqueName: \"kubernetes.io/projected/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-kube-api-access-4ztvw\") pod \"istiod-openshift-gateway-7876bfcb74-wg2xp\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:00:41.349319 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.349255    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-wg2xp\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:00:41.349620 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.349594    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-wg2xp\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:00:41.351259 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.351212    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-wg2xp\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:00:41.351353 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.351269    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-wg2xp\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:00:41.351425 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.351398    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-wg2xp\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:00:41.351425 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.351403    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-wg2xp\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:00:41.373578 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.373554    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ztvw\" (UniqueName: \"kubernetes.io/projected/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-kube-api-access-4ztvw\") pod \"istiod-openshift-gateway-7876bfcb74-wg2xp\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:00:41.373683 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.373650    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-wg2xp\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:00:41.373760 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.373742    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn"]
Oct 30 04:00:41.376408 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:00:41.376387    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d043bc_6a33_439f_b75e_2716b30bce34.slice/crio-42bdc68ce5177e67336732aa2edfe35aeb47a35eaa0fae6e70579e2c88b86046 WatchSource:0}: Error finding container 42bdc68ce5177e67336732aa2edfe35aeb47a35eaa0fae6e70579e2c88b86046: Status 404 returned error can't find the container with id 42bdc68ce5177e67336732aa2edfe35aeb47a35eaa0fae6e70579e2c88b86046
Oct 30 04:00:41.454574 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.454552    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:00:41.497378 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.497347    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7"]
Oct 30 04:00:41.503255 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:00:41.501911    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeafd1bc_0c7a_4268_a363_7c12399045b3.slice/crio-ac200777a4296a0ca6e33ec2213a960f0ffc4ba2f100f7644b43feb80968b236 WatchSource:0}: Error finding container ac200777a4296a0ca6e33ec2213a960f0ffc4ba2f100f7644b43feb80968b236: Status 404 returned error can't find the container with id ac200777a4296a0ca6e33ec2213a960f0ffc4ba2f100f7644b43feb80968b236
Oct 30 04:00:41.752609 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:41.752586    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"]
Oct 30 04:00:41.754286 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:00:41.754260    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee8ebc1_e44c_447c_95b9_566ee210c2c1.slice/crio-5bf1326fe0e925f3b4758186edde1a30c98487bd6de9a804ca74beaaf7eab61a WatchSource:0}: Error finding container 5bf1326fe0e925f3b4758186edde1a30c98487bd6de9a804ca74beaaf7eab61a: Status 404 returned error can't find the container with id 5bf1326fe0e925f3b4758186edde1a30c98487bd6de9a804ca74beaaf7eab61a
Oct 30 04:00:42.256340 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:42.256306    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn" event={"ID":"73d043bc-6a33-439f-b75e-2716b30bce34","Type":"ContainerStarted","Data":"42bdc68ce5177e67336732aa2edfe35aeb47a35eaa0fae6e70579e2c88b86046"}
Oct 30 04:00:42.257753 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:42.257725    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp" event={"ID":"8ee8ebc1-e44c-447c-95b9-566ee210c2c1","Type":"ContainerStarted","Data":"5bf1326fe0e925f3b4758186edde1a30c98487bd6de9a804ca74beaaf7eab61a"}
Oct 30 04:00:42.258894 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:42.258865    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7" event={"ID":"aeafd1bc-0c7a-4268-a363-7c12399045b3","Type":"ContainerStarted","Data":"ac200777a4296a0ca6e33ec2213a960f0ffc4ba2f100f7644b43feb80968b236"}
Oct 30 04:00:44.973711 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:44.973678    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:00:44.974106 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:44.973758    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:00:45.274110 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:45.274064    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp" event={"ID":"8ee8ebc1-e44c-447c-95b9-566ee210c2c1","Type":"ContainerStarted","Data":"d7e053cfa77bd4fbdee90500dd99e39400e8ae4a0d84620c031db0f5189c3ba3"}
Oct 30 04:00:45.274287 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:45.274201    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:00:45.324853 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:45.324800    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp" podStartSLOduration=1.108232176 podStartE2EDuration="4.324781223s" totalImagesPullingTime="3.216549047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:45.323667884 +0000 UTC m=+164.736063458" watchObservedRunningTime="2026-10-30 04:00:45.324781223 +0000 UTC m=+164.737176799"
Oct 30 04:00:46.280573 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:46.280536    4874 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-wg2xp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 04:00:46.281075 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:46.280603    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp" podUID="8ee8ebc1-e44c-447c-95b9-566ee210c2c1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:00:48.287275 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:48.287220    4874 generic.go:356] "Generic (PLEG): container finished" podID="aeafd1bc-0c7a-4268-a363-7c12399045b3" containerID="e299bd92bfaf5fde5804d67cf85004df182a612178070c32cce68acac9f38a8a" exitCode=0
Oct 30 04:00:48.287671 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:48.287309    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7" event={"ID":"aeafd1bc-0c7a-4268-a363-7c12399045b3","Type":"ContainerDied","Data":"e299bd92bfaf5fde5804d67cf85004df182a612178070c32cce68acac9f38a8a"}
Oct 30 04:00:48.288810 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:48.288790    4874 generic.go:356] "Generic (PLEG): container finished" podID="73d043bc-6a33-439f-b75e-2716b30bce34" containerID="e9c896600c245abe2a47488c15953a86af95454c6ba43db20058810a12a32569" exitCode=0
Oct 30 04:00:48.288891 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:48.288872    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn" event={"ID":"73d043bc-6a33-439f-b75e-2716b30bce34","Type":"ContainerDied","Data":"e9c896600c245abe2a47488c15953a86af95454c6ba43db20058810a12a32569"}
Oct 30 04:00:49.279737 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:49.279711    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:00:49.295037 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:49.295000    4874 generic.go:356] "Generic (PLEG): container finished" podID="aeafd1bc-0c7a-4268-a363-7c12399045b3" containerID="6e8b49cb28e100294fb15393f33d65830cdda0d47ef30042b25c90215c6c0d04" exitCode=0
Oct 30 04:00:49.295037 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:49.295038    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7" event={"ID":"aeafd1bc-0c7a-4268-a363-7c12399045b3","Type":"ContainerDied","Data":"6e8b49cb28e100294fb15393f33d65830cdda0d47ef30042b25c90215c6c0d04"}
Oct 30 04:00:51.304910 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:51.304867    4874 generic.go:356] "Generic (PLEG): container finished" podID="73d043bc-6a33-439f-b75e-2716b30bce34" containerID="e8166e7c57b73726f7a97cf876bfdb653912ebe3b1f59e8fd88c590b4d5c7bc0" exitCode=0
Oct 30 04:00:51.305435 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:51.304922    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn" event={"ID":"73d043bc-6a33-439f-b75e-2716b30bce34","Type":"ContainerDied","Data":"e8166e7c57b73726f7a97cf876bfdb653912ebe3b1f59e8fd88c590b4d5c7bc0"}
Oct 30 04:00:55.615383 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:00:55.615316    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:00:56.327932 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:56.327850    4874 generic.go:356] "Generic (PLEG): container finished" podID="aeafd1bc-0c7a-4268-a363-7c12399045b3" containerID="c270f751307cdca28944ff7244178ef4753958a3630ae6a2485520371b520c1d" exitCode=0
Oct 30 04:00:56.327932 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:56.327919    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7" event={"ID":"aeafd1bc-0c7a-4268-a363-7c12399045b3","Type":"ContainerDied","Data":"c270f751307cdca28944ff7244178ef4753958a3630ae6a2485520371b520c1d"}
Oct 30 04:00:57.453409 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:57.453385    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7"
Oct 30 04:00:57.590745 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:57.590665    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aeafd1bc-0c7a-4268-a363-7c12399045b3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/aeafd1bc-0c7a-4268-a363-7c12399045b3-bundle\") pod \"aeafd1bc-0c7a-4268-a363-7c12399045b3\" (UID: \"aeafd1bc-0c7a-4268-a363-7c12399045b3\") "
Oct 30 04:00:57.590745 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:57.590730    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeafd1bc-0c7a-4268-a363-7c12399045b3-kube-api-access-2k9n8\" (UniqueName: \"kubernetes.io/projected/aeafd1bc-0c7a-4268-a363-7c12399045b3-kube-api-access-2k9n8\") pod \"aeafd1bc-0c7a-4268-a363-7c12399045b3\" (UID: \"aeafd1bc-0c7a-4268-a363-7c12399045b3\") "
Oct 30 04:00:57.590933 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:57.590784    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aeafd1bc-0c7a-4268-a363-7c12399045b3-util\" (UniqueName: \"kubernetes.io/empty-dir/aeafd1bc-0c7a-4268-a363-7c12399045b3-util\") pod \"aeafd1bc-0c7a-4268-a363-7c12399045b3\" (UID: \"aeafd1bc-0c7a-4268-a363-7c12399045b3\") "
Oct 30 04:00:57.591369 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:57.591340    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeafd1bc-0c7a-4268-a363-7c12399045b3-bundle" pod "aeafd1bc-0c7a-4268-a363-7c12399045b3" (UID: "aeafd1bc-0c7a-4268-a363-7c12399045b3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:57.593029 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:57.593008    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeafd1bc-0c7a-4268-a363-7c12399045b3-kube-api-access-2k9n8" pod "aeafd1bc-0c7a-4268-a363-7c12399045b3" (UID: "aeafd1bc-0c7a-4268-a363-7c12399045b3"). InnerVolumeSpecName "kube-api-access-2k9n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:57.595021 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:57.594997    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeafd1bc-0c7a-4268-a363-7c12399045b3-util" pod "aeafd1bc-0c7a-4268-a363-7c12399045b3" (UID: "aeafd1bc-0c7a-4268-a363-7c12399045b3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:57.691338 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:57.691297    4874 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aeafd1bc-0c7a-4268-a363-7c12399045b3-bundle\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:00:57.691338 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:57.691330    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k9n8\" (UniqueName: \"kubernetes.io/projected/aeafd1bc-0c7a-4268-a363-7c12399045b3-kube-api-access-2k9n8\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:00:57.691536 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:57.691344    4874 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aeafd1bc-0c7a-4268-a363-7c12399045b3-util\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:00:58.335848 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:58.335819    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7"
Oct 30 04:00:58.336025 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:58.335817    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dv6w7" event={"ID":"aeafd1bc-0c7a-4268-a363-7c12399045b3","Type":"ContainerDied","Data":"ac200777a4296a0ca6e33ec2213a960f0ffc4ba2f100f7644b43feb80968b236"}
Oct 30 04:00:58.336025 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:58.335939    4874 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac200777a4296a0ca6e33ec2213a960f0ffc4ba2f100f7644b43feb80968b236"
Oct 30 04:00:58.337953 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:58.337927    4874 generic.go:356] "Generic (PLEG): container finished" podID="73d043bc-6a33-439f-b75e-2716b30bce34" containerID="c723353af043c11f91ff65269a105979b2ea1b881554c62a6af3b994f8be888f" exitCode=0
Oct 30 04:00:58.338090 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:58.337977    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn" event={"ID":"73d043bc-6a33-439f-b75e-2716b30bce34","Type":"ContainerDied","Data":"c723353af043c11f91ff65269a105979b2ea1b881554c62a6af3b994f8be888f"}
Oct 30 04:00:59.453371 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:59.453347    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn"
Oct 30 04:00:59.604000 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:59.603928    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73d043bc-6a33-439f-b75e-2716b30bce34-bundle\" (UniqueName: \"kubernetes.io/empty-dir/73d043bc-6a33-439f-b75e-2716b30bce34-bundle\") pod \"73d043bc-6a33-439f-b75e-2716b30bce34\" (UID: \"73d043bc-6a33-439f-b75e-2716b30bce34\") "
Oct 30 04:00:59.604000 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:59.603965    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73d043bc-6a33-439f-b75e-2716b30bce34-kube-api-access-trvnp\" (UniqueName: \"kubernetes.io/projected/73d043bc-6a33-439f-b75e-2716b30bce34-kube-api-access-trvnp\") pod \"73d043bc-6a33-439f-b75e-2716b30bce34\" (UID: \"73d043bc-6a33-439f-b75e-2716b30bce34\") "
Oct 30 04:00:59.604212 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:59.604002    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73d043bc-6a33-439f-b75e-2716b30bce34-util\" (UniqueName: \"kubernetes.io/empty-dir/73d043bc-6a33-439f-b75e-2716b30bce34-util\") pod \"73d043bc-6a33-439f-b75e-2716b30bce34\" (UID: \"73d043bc-6a33-439f-b75e-2716b30bce34\") "
Oct 30 04:00:59.604634 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:59.604609    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73d043bc-6a33-439f-b75e-2716b30bce34-bundle" pod "73d043bc-6a33-439f-b75e-2716b30bce34" (UID: "73d043bc-6a33-439f-b75e-2716b30bce34"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:59.605993 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:59.605971    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d043bc-6a33-439f-b75e-2716b30bce34-kube-api-access-trvnp" pod "73d043bc-6a33-439f-b75e-2716b30bce34" (UID: "73d043bc-6a33-439f-b75e-2716b30bce34"). InnerVolumeSpecName "kube-api-access-trvnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:59.609468 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:59.609444    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73d043bc-6a33-439f-b75e-2716b30bce34-util" pod "73d043bc-6a33-439f-b75e-2716b30bce34" (UID: "73d043bc-6a33-439f-b75e-2716b30bce34"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:59.704745 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:59.704714    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trvnp\" (UniqueName: \"kubernetes.io/projected/73d043bc-6a33-439f-b75e-2716b30bce34-kube-api-access-trvnp\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:00:59.704745 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:59.704741    4874 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73d043bc-6a33-439f-b75e-2716b30bce34-util\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:00:59.704745 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:00:59.704750    4874 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73d043bc-6a33-439f-b75e-2716b30bce34-bundle\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:01:00.346698 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:00.346671    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn"
Oct 30 04:01:00.346868 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:00.346673    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7h5zn" event={"ID":"73d043bc-6a33-439f-b75e-2716b30bce34","Type":"ContainerDied","Data":"42bdc68ce5177e67336732aa2edfe35aeb47a35eaa0fae6e70579e2c88b86046"}
Oct 30 04:01:00.346868 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:00.346782    4874 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42bdc68ce5177e67336732aa2edfe35aeb47a35eaa0fae6e70579e2c88b86046"
Oct 30 04:01:23.923093 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:23.923052    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5"]
Oct 30 04:01:23.923727 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:23.923441    4874 cpu_manager.go:414] "containerMap: removing container" podUID="aeafd1bc-0c7a-4268-a363-7c12399045b3" containerName="util"
Oct 30 04:01:23.923727 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:23.923459    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeafd1bc-0c7a-4268-a363-7c12399045b3" containerName="util"
Oct 30 04:01:23.923727 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:23.923484    4874 cpu_manager.go:414] "containerMap: removing container" podUID="aeafd1bc-0c7a-4268-a363-7c12399045b3" containerName="pull"
Oct 30 04:01:23.923727 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:23.923491    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeafd1bc-0c7a-4268-a363-7c12399045b3" containerName="pull"
Oct 30 04:01:23.923727 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:23.923501    4874 cpu_manager.go:414] "containerMap: removing container" podUID="73d043bc-6a33-439f-b75e-2716b30bce34" containerName="pull"
Oct 30 04:01:23.923727 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:23.923508    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d043bc-6a33-439f-b75e-2716b30bce34" containerName="pull"
Oct 30 04:01:23.923727 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:23.923517    4874 cpu_manager.go:414] "containerMap: removing container" podUID="aeafd1bc-0c7a-4268-a363-7c12399045b3" containerName="extract"
Oct 30 04:01:23.923727 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:23.923523    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeafd1bc-0c7a-4268-a363-7c12399045b3" containerName="extract"
Oct 30 04:01:23.923727 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:23.923541    4874 cpu_manager.go:414] "containerMap: removing container" podUID="73d043bc-6a33-439f-b75e-2716b30bce34" containerName="util"
Oct 30 04:01:23.923727 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:23.923547    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d043bc-6a33-439f-b75e-2716b30bce34" containerName="util"
Oct 30 04:01:23.923727 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:23.923565    4874 cpu_manager.go:414] "containerMap: removing container" podUID="73d043bc-6a33-439f-b75e-2716b30bce34" containerName="extract"
Oct 30 04:01:23.923727 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:23.923573    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d043bc-6a33-439f-b75e-2716b30bce34" containerName="extract"
Oct 30 04:01:23.923727 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:23.923645    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeafd1bc-0c7a-4268-a363-7c12399045b3" containerName="extract"
Oct 30 04:01:23.923727 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:23.923658    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d043bc-6a33-439f-b75e-2716b30bce34" containerName="extract"
Oct 30 04:01:23.925721 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:23.925700    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5"
Oct 30 04:01:23.957455 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:23.957430    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5"]
Oct 30 04:01:24.003921 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.003894    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-8srh5\""
Oct 30 04:01:24.003921 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.003899    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 04:01:24.003921 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.003898    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 04:01:24.004128 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.003905    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 04:01:24.004128 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.003970    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 04:01:24.004128 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.003975    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 04:01:24.069501 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.069473    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/412d9d4d-1c11-4b13-b582-1e3c839a4ab2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-9tgc5\" (UID: \"412d9d4d-1c11-4b13-b582-1e3c839a4ab2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5"
Oct 30 04:01:24.069630 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.069513    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/412d9d4d-1c11-4b13-b582-1e3c839a4ab2-tmp\") pod \"opendatahub-operator-controller-manager-b54c6978-9tgc5\" (UID: \"412d9d4d-1c11-4b13-b582-1e3c839a4ab2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5"
Oct 30 04:01:24.069630 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.069540    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/412d9d4d-1c11-4b13-b582-1e3c839a4ab2-metrics-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-9tgc5\" (UID: \"412d9d4d-1c11-4b13-b582-1e3c839a4ab2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5"
Oct 30 04:01:24.069630 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.069589    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzcrr\" (UniqueName: \"kubernetes.io/projected/412d9d4d-1c11-4b13-b582-1e3c839a4ab2-kube-api-access-wzcrr\") pod \"opendatahub-operator-controller-manager-b54c6978-9tgc5\" (UID: \"412d9d4d-1c11-4b13-b582-1e3c839a4ab2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5"
Oct 30 04:01:24.069743 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.069692    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/412d9d4d-1c11-4b13-b582-1e3c839a4ab2-manifests\") pod \"opendatahub-operator-controller-manager-b54c6978-9tgc5\" (UID: \"412d9d4d-1c11-4b13-b582-1e3c839a4ab2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5"
Oct 30 04:01:24.069743 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.069717    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/412d9d4d-1c11-4b13-b582-1e3c839a4ab2-webhook-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-9tgc5\" (UID: \"412d9d4d-1c11-4b13-b582-1e3c839a4ab2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5"
Oct 30 04:01:24.170888 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.170859    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/412d9d4d-1c11-4b13-b582-1e3c839a4ab2-metrics-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-9tgc5\" (UID: \"412d9d4d-1c11-4b13-b582-1e3c839a4ab2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5"
Oct 30 04:01:24.170888 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.170888    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzcrr\" (UniqueName: \"kubernetes.io/projected/412d9d4d-1c11-4b13-b582-1e3c839a4ab2-kube-api-access-wzcrr\") pod \"opendatahub-operator-controller-manager-b54c6978-9tgc5\" (UID: \"412d9d4d-1c11-4b13-b582-1e3c839a4ab2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5"
Oct 30 04:01:24.171044 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.170909    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/412d9d4d-1c11-4b13-b582-1e3c839a4ab2-manifests\") pod \"opendatahub-operator-controller-manager-b54c6978-9tgc5\" (UID: \"412d9d4d-1c11-4b13-b582-1e3c839a4ab2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5"
Oct 30 04:01:24.171044 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.170926    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/412d9d4d-1c11-4b13-b582-1e3c839a4ab2-webhook-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-9tgc5\" (UID: \"412d9d4d-1c11-4b13-b582-1e3c839a4ab2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5"
Oct 30 04:01:24.171044 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.170951    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/412d9d4d-1c11-4b13-b582-1e3c839a4ab2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-9tgc5\" (UID: \"412d9d4d-1c11-4b13-b582-1e3c839a4ab2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5"
Oct 30 04:01:24.171044 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.170983    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/412d9d4d-1c11-4b13-b582-1e3c839a4ab2-tmp\") pod \"opendatahub-operator-controller-manager-b54c6978-9tgc5\" (UID: \"412d9d4d-1c11-4b13-b582-1e3c839a4ab2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5"
Oct 30 04:01:24.171379 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.171346    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/412d9d4d-1c11-4b13-b582-1e3c839a4ab2-manifests\") pod \"opendatahub-operator-controller-manager-b54c6978-9tgc5\" (UID: \"412d9d4d-1c11-4b13-b582-1e3c839a4ab2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5"
Oct 30 04:01:24.173273 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.173198    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/412d9d4d-1c11-4b13-b582-1e3c839a4ab2-tmp\") pod \"opendatahub-operator-controller-manager-b54c6978-9tgc5\" (UID: \"412d9d4d-1c11-4b13-b582-1e3c839a4ab2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5"
Oct 30 04:01:24.173384 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.173357    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/412d9d4d-1c11-4b13-b582-1e3c839a4ab2-webhook-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-9tgc5\" (UID: \"412d9d4d-1c11-4b13-b582-1e3c839a4ab2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5"
Oct 30 04:01:24.173499 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.173480    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/412d9d4d-1c11-4b13-b582-1e3c839a4ab2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-9tgc5\" (UID: \"412d9d4d-1c11-4b13-b582-1e3c839a4ab2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5"
Oct 30 04:01:24.173558 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.173501    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/412d9d4d-1c11-4b13-b582-1e3c839a4ab2-metrics-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-9tgc5\" (UID: \"412d9d4d-1c11-4b13-b582-1e3c839a4ab2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5"
Oct 30 04:01:24.199553 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.199531    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzcrr\" (UniqueName: \"kubernetes.io/projected/412d9d4d-1c11-4b13-b582-1e3c839a4ab2-kube-api-access-wzcrr\") pod \"opendatahub-operator-controller-manager-b54c6978-9tgc5\" (UID: \"412d9d4d-1c11-4b13-b582-1e3c839a4ab2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5"
Oct 30 04:01:24.237417 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.237391    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5"
Oct 30 04:01:24.678509 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:24.678481    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5"]
Oct 30 04:01:24.680274 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:01:24.680251    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412d9d4d_1c11_4b13_b582_1e3c839a4ab2.slice/crio-5fd6f6517b50083cfc0b5e428749a448496671c1162c9cecb157a25d31709dd8 WatchSource:0}: Error finding container 5fd6f6517b50083cfc0b5e428749a448496671c1162c9cecb157a25d31709dd8: Status 404 returned error can't find the container with id 5fd6f6517b50083cfc0b5e428749a448496671c1162c9cecb157a25d31709dd8
Oct 30 04:01:25.432599 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:25.432559    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5" event={"ID":"412d9d4d-1c11-4b13-b582-1e3c839a4ab2","Type":"ContainerStarted","Data":"5fd6f6517b50083cfc0b5e428749a448496671c1162c9cecb157a25d31709dd8"}
Oct 30 04:01:25.617777 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:01:25.617730    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:01:28.444812 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:28.444779    4874 generic.go:356] "Generic (PLEG): container finished" podID="412d9d4d-1c11-4b13-b582-1e3c839a4ab2" containerID="ed3c4a2b2c45a514166fe5dda3274218b701dcc57950b6a05f9d5fa1f6fc9ea6" exitCode=0
Oct 30 04:01:28.445204 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:28.444817    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5" event={"ID":"412d9d4d-1c11-4b13-b582-1e3c839a4ab2","Type":"ContainerDied","Data":"ed3c4a2b2c45a514166fe5dda3274218b701dcc57950b6a05f9d5fa1f6fc9ea6"}
Oct 30 04:01:29.449502 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:29.449460    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5" event={"ID":"412d9d4d-1c11-4b13-b582-1e3c839a4ab2","Type":"ContainerStarted","Data":"6dc7a6e7a53dd522e8992b5febd1ce35e726aab8c82e1c217b8f22a138ababf3"}
Oct 30 04:01:29.449925 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:29.449587    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5"
Oct 30 04:01:29.536392 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:29.536345    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5" podStartSLOduration=2.948012029 podStartE2EDuration="6.536331781s" totalImagesPullingTime="3.16552551s" totalInitContainerRuntime="422.794242ms" isStatefulPod=false podCreationTimestamp="2026-10-30 04:01:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:29.524349631 +0000 UTC m=+208.936745203" watchObservedRunningTime="2026-10-30 04:01:29.536331781 +0000 UTC m=+208.948727355"
Oct 30 04:01:37.591370 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:37.591337    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-wf5lc"]
Oct 30 04:01:37.595301 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:37.595280    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-wf5lc"
Oct 30 04:01:37.610844 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:37.610818    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-442ts\""
Oct 30 04:01:37.622266 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:37.622229    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 04:01:37.622416 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:37.622401    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 04:01:37.623381 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:37.623364    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-wf5lc"]
Oct 30 04:01:37.668217 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:37.668188    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2295f348-4c99-49f5-b81c-24072dfa520d-bound-sa-token\") pod \"cert-manager-597b95cb84-wf5lc\" (UID: \"2295f348-4c99-49f5-b81c-24072dfa520d\") " pod="cert-manager/cert-manager-597b95cb84-wf5lc"
Oct 30 04:01:37.668345 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:37.668221    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-559z6\" (UniqueName: \"kubernetes.io/projected/2295f348-4c99-49f5-b81c-24072dfa520d-kube-api-access-559z6\") pod \"cert-manager-597b95cb84-wf5lc\" (UID: \"2295f348-4c99-49f5-b81c-24072dfa520d\") " pod="cert-manager/cert-manager-597b95cb84-wf5lc"
Oct 30 04:01:37.769394 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:37.769365    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2295f348-4c99-49f5-b81c-24072dfa520d-bound-sa-token\") pod \"cert-manager-597b95cb84-wf5lc\" (UID: \"2295f348-4c99-49f5-b81c-24072dfa520d\") " pod="cert-manager/cert-manager-597b95cb84-wf5lc"
Oct 30 04:01:37.769531 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:37.769397    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-559z6\" (UniqueName: \"kubernetes.io/projected/2295f348-4c99-49f5-b81c-24072dfa520d-kube-api-access-559z6\") pod \"cert-manager-597b95cb84-wf5lc\" (UID: \"2295f348-4c99-49f5-b81c-24072dfa520d\") " pod="cert-manager/cert-manager-597b95cb84-wf5lc"
Oct 30 04:01:37.794837 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:37.794805    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-559z6\" (UniqueName: \"kubernetes.io/projected/2295f348-4c99-49f5-b81c-24072dfa520d-kube-api-access-559z6\") pod \"cert-manager-597b95cb84-wf5lc\" (UID: \"2295f348-4c99-49f5-b81c-24072dfa520d\") " pod="cert-manager/cert-manager-597b95cb84-wf5lc"
Oct 30 04:01:37.795055 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:37.795038    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2295f348-4c99-49f5-b81c-24072dfa520d-bound-sa-token\") pod \"cert-manager-597b95cb84-wf5lc\" (UID: \"2295f348-4c99-49f5-b81c-24072dfa520d\") " pod="cert-manager/cert-manager-597b95cb84-wf5lc"
Oct 30 04:01:37.916868 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:37.916808    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-wf5lc"
Oct 30 04:01:38.218301 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:38.218278    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-wf5lc"]
Oct 30 04:01:38.220552 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:01:38.220522    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2295f348_4c99_49f5_b81c_24072dfa520d.slice/crio-33957f898ffa8717b94322d8a5d2a8dea60cfda768fe4e10e89549b792e550cd WatchSource:0}: Error finding container 33957f898ffa8717b94322d8a5d2a8dea60cfda768fe4e10e89549b792e550cd: Status 404 returned error can't find the container with id 33957f898ffa8717b94322d8a5d2a8dea60cfda768fe4e10e89549b792e550cd
Oct 30 04:01:38.480984 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:38.480956    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-wf5lc" event={"ID":"2295f348-4c99-49f5-b81c-24072dfa520d","Type":"ContainerStarted","Data":"33957f898ffa8717b94322d8a5d2a8dea60cfda768fe4e10e89549b792e550cd"}
Oct 30 04:01:40.455526 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:40.455495    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-9tgc5"
Oct 30 04:01:44.503106 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:44.503068    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-wf5lc" event={"ID":"2295f348-4c99-49f5-b81c-24072dfa520d","Type":"ContainerStarted","Data":"b105b334765fed8a351d2800c66679dc3c22a610701c4ecd2a15f8c11600cb0a"}
Oct 30 04:01:44.544194 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:44.544143    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-wf5lc" podStartSLOduration=1.81514938 podStartE2EDuration="7.544132694s" totalImagesPullingTime="5.728983314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:44.543583628 +0000 UTC m=+223.955979198" watchObservedRunningTime="2026-10-30 04:01:44.544132694 +0000 UTC m=+223.956528266"
Oct 30 04:01:50.022862 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:50.022821    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vmffb"]
Oct 30 04:01:50.025340 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:50.025314    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vmffb"
Oct 30 04:01:50.040667 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:50.040645    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 04:01:50.052054 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:50.052028    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cn6xf\""
Oct 30 04:01:50.052221 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:50.052205    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 04:01:50.052320 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:50.052217    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 04:01:50.052320 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:50.052218    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 04:01:50.053169 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:50.053154    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vmffb"]
Oct 30 04:01:50.062536 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:50.062518    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6q5d\" (UniqueName: \"kubernetes.io/projected/93fe6d14-eca6-492a-9237-75362698993f-kube-api-access-m6q5d\") pod \"kuadrant-console-plugin-66c4fb575d-vmffb\" (UID: \"93fe6d14-eca6-492a-9237-75362698993f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vmffb"
Oct 30 04:01:50.062587 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:50.062561    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/93fe6d14-eca6-492a-9237-75362698993f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vmffb\" (UID: \"93fe6d14-eca6-492a-9237-75362698993f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vmffb"
Oct 30 04:01:50.062667 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:50.062648    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/93fe6d14-eca6-492a-9237-75362698993f-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vmffb\" (UID: \"93fe6d14-eca6-492a-9237-75362698993f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vmffb"
Oct 30 04:01:50.163571 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:50.163542    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/93fe6d14-eca6-492a-9237-75362698993f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vmffb\" (UID: \"93fe6d14-eca6-492a-9237-75362698993f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vmffb"
Oct 30 04:01:50.163686 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:50.163588    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/93fe6d14-eca6-492a-9237-75362698993f-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vmffb\" (UID: \"93fe6d14-eca6-492a-9237-75362698993f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vmffb"
Oct 30 04:01:50.163686 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:50.163620    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6q5d\" (UniqueName: \"kubernetes.io/projected/93fe6d14-eca6-492a-9237-75362698993f-kube-api-access-m6q5d\") pod \"kuadrant-console-plugin-66c4fb575d-vmffb\" (UID: \"93fe6d14-eca6-492a-9237-75362698993f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vmffb"
Oct 30 04:01:50.164304 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:50.164286    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/93fe6d14-eca6-492a-9237-75362698993f-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vmffb\" (UID: \"93fe6d14-eca6-492a-9237-75362698993f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vmffb"
Oct 30 04:01:50.165891 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:50.165869    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/93fe6d14-eca6-492a-9237-75362698993f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vmffb\" (UID: \"93fe6d14-eca6-492a-9237-75362698993f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vmffb"
Oct 30 04:01:50.185921 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:50.185894    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6q5d\" (UniqueName: \"kubernetes.io/projected/93fe6d14-eca6-492a-9237-75362698993f-kube-api-access-m6q5d\") pod \"kuadrant-console-plugin-66c4fb575d-vmffb\" (UID: \"93fe6d14-eca6-492a-9237-75362698993f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vmffb"
Oct 30 04:01:50.334967 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:50.334910    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vmffb"
Oct 30 04:01:50.628587 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:50.628528    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vmffb"]
Oct 30 04:01:50.630738 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:01:50.630711    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93fe6d14_eca6_492a_9237_75362698993f.slice/crio-b33339b840ec143fe96074290d5ebe3fce3e451f171bc9eae737bc27ee90f05e WatchSource:0}: Error finding container b33339b840ec143fe96074290d5ebe3fce3e451f171bc9eae737bc27ee90f05e: Status 404 returned error can't find the container with id b33339b840ec143fe96074290d5ebe3fce3e451f171bc9eae737bc27ee90f05e
Oct 30 04:01:51.531468 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:51.531423    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vmffb" event={"ID":"93fe6d14-eca6-492a-9237-75362698993f","Type":"ContainerStarted","Data":"b33339b840ec143fe96074290d5ebe3fce3e451f171bc9eae737bc27ee90f05e"}
Oct 30 04:01:55.547940 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:55.547902    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vmffb" event={"ID":"93fe6d14-eca6-492a-9237-75362698993f","Type":"ContainerStarted","Data":"9365a299e4807d6d22e105713aa42bf7a337b7b423e332675981523969dfb0af"}
Oct 30 04:01:55.589447 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:55.589394    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vmffb" podStartSLOduration=2.120025746 podStartE2EDuration="6.589379004s" totalImagesPullingTime="4.469353258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:01:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:55.588420584 +0000 UTC m=+235.000816157" watchObservedRunningTime="2026-10-30 04:01:55.589379004 +0000 UTC m=+235.001774574"
Oct 30 04:01:55.610675 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:01:55.610638    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:01:59.535333 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:59.535295    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-685cdf9dcb-sgtk5"]
Oct 30 04:01:59.540857 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:59.540837    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-sgtk5"
Oct 30 04:01:59.568602 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:59.568573    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-685cdf9dcb-sgtk5"]
Oct 30 04:01:59.602098 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:59.602072    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 04:01:59.609727 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:59.609704    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xs8bw\""
Oct 30 04:01:59.609867 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:59.609784    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 04:01:59.636199 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:59.636171    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zgfc\" (UniqueName: \"kubernetes.io/projected/38243efb-429f-450b-a62a-6b55c62330a6-kube-api-access-5zgfc\") pod \"kube-auth-proxy-685cdf9dcb-sgtk5\" (UID: \"38243efb-429f-450b-a62a-6b55c62330a6\") " pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-sgtk5"
Oct 30 04:01:59.636339 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:59.636209    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38243efb-429f-450b-a62a-6b55c62330a6-tmp\") pod \"kube-auth-proxy-685cdf9dcb-sgtk5\" (UID: \"38243efb-429f-450b-a62a-6b55c62330a6\") " pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-sgtk5"
Oct 30 04:01:59.636380 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:59.636335    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38243efb-429f-450b-a62a-6b55c62330a6-tls-certs\") pod \"kube-auth-proxy-685cdf9dcb-sgtk5\" (UID: \"38243efb-429f-450b-a62a-6b55c62330a6\") " pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-sgtk5"
Oct 30 04:01:59.737246 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:59.737217    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38243efb-429f-450b-a62a-6b55c62330a6-tls-certs\") pod \"kube-auth-proxy-685cdf9dcb-sgtk5\" (UID: \"38243efb-429f-450b-a62a-6b55c62330a6\") " pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-sgtk5"
Oct 30 04:01:59.737408 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:59.737286    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zgfc\" (UniqueName: \"kubernetes.io/projected/38243efb-429f-450b-a62a-6b55c62330a6-kube-api-access-5zgfc\") pod \"kube-auth-proxy-685cdf9dcb-sgtk5\" (UID: \"38243efb-429f-450b-a62a-6b55c62330a6\") " pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-sgtk5"
Oct 30 04:01:59.737408 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:59.737323    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38243efb-429f-450b-a62a-6b55c62330a6-tmp\") pod \"kube-auth-proxy-685cdf9dcb-sgtk5\" (UID: \"38243efb-429f-450b-a62a-6b55c62330a6\") " pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-sgtk5"
Oct 30 04:01:59.739608 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:59.739583    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38243efb-429f-450b-a62a-6b55c62330a6-tmp\") pod \"kube-auth-proxy-685cdf9dcb-sgtk5\" (UID: \"38243efb-429f-450b-a62a-6b55c62330a6\") " pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-sgtk5"
Oct 30 04:01:59.739728 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:59.739710    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38243efb-429f-450b-a62a-6b55c62330a6-tls-certs\") pod \"kube-auth-proxy-685cdf9dcb-sgtk5\" (UID: \"38243efb-429f-450b-a62a-6b55c62330a6\") " pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-sgtk5"
Oct 30 04:01:59.790173 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:59.790095    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zgfc\" (UniqueName: \"kubernetes.io/projected/38243efb-429f-450b-a62a-6b55c62330a6-kube-api-access-5zgfc\") pod \"kube-auth-proxy-685cdf9dcb-sgtk5\" (UID: \"38243efb-429f-450b-a62a-6b55c62330a6\") " pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-sgtk5"
Oct 30 04:01:59.851390 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:01:59.851356    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-sgtk5"
Oct 30 04:02:00.146508 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:00.146477    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-685cdf9dcb-sgtk5"]
Oct 30 04:02:00.148758 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:02:00.148736    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38243efb_429f_450b_a62a_6b55c62330a6.slice/crio-2f08944955aecb5321789302fd10bb1b75e82a602a285902ea67ee0550566ebf WatchSource:0}: Error finding container 2f08944955aecb5321789302fd10bb1b75e82a602a285902ea67ee0550566ebf: Status 404 returned error can't find the container with id 2f08944955aecb5321789302fd10bb1b75e82a602a285902ea67ee0550566ebf
Oct 30 04:02:00.566956 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:00.566917    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-sgtk5" event={"ID":"38243efb-429f-450b-a62a-6b55c62330a6","Type":"ContainerStarted","Data":"2f08944955aecb5321789302fd10bb1b75e82a602a285902ea67ee0550566ebf"}
Oct 30 04:02:02.513105 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:02.513070    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 04:02:03.580709 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:03.580677    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-sgtk5" event={"ID":"38243efb-429f-450b-a62a-6b55c62330a6","Type":"ContainerStarted","Data":"1d875f362853b410eee0fdacfb08c5702b61b22896497639fc37d1f2622e2d05"}
Oct 30 04:02:03.619991 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:03.619942    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-685cdf9dcb-sgtk5" podStartSLOduration=2.27099592 podStartE2EDuration="4.619930581s" totalImagesPullingTime="2.348934661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:01:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:02:03.618779851 +0000 UTC m=+243.031175424" watchObservedRunningTime="2026-10-30 04:02:03.619930581 +0000 UTC m=+243.032326153"
Oct 30 04:02:25.608957 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:02:25.608919    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:02:55.607675 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:02:55.607635    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:02:58.407065 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:58.407030    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"]
Oct 30 04:02:58.409282 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:58.409264    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"
Oct 30 04:02:58.438824 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:58.438798    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"]
Oct 30 04:02:58.450898 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:58.450870    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e6ab204-9f50-4bd2-8b0e-0044278d820d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pv6t\" (UID: \"6e6ab204-9f50-4bd2-8b0e-0044278d820d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"
Oct 30 04:02:58.451008 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:58.450916    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e6ab204-9f50-4bd2-8b0e-0044278d820d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pv6t\" (UID: \"6e6ab204-9f50-4bd2-8b0e-0044278d820d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"
Oct 30 04:02:58.451008 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:58.450975    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf2tn\" (UniqueName: \"kubernetes.io/projected/6e6ab204-9f50-4bd2-8b0e-0044278d820d-kube-api-access-pf2tn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pv6t\" (UID: \"6e6ab204-9f50-4bd2-8b0e-0044278d820d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"
Oct 30 04:02:58.451122 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:58.451105    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6e6ab204-9f50-4bd2-8b0e-0044278d820d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pv6t\" (UID: \"6e6ab204-9f50-4bd2-8b0e-0044278d820d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"
Oct 30 04:02:58.451162 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:58.451143    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6e6ab204-9f50-4bd2-8b0e-0044278d820d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pv6t\" (UID: \"6e6ab204-9f50-4bd2-8b0e-0044278d820d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"
Oct 30 04:02:58.451226 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:58.451213    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6e6ab204-9f50-4bd2-8b0e-0044278d820d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pv6t\" (UID: \"6e6ab204-9f50-4bd2-8b0e-0044278d820d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"
Oct 30 04:02:58.451307 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:58.451255    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e6ab204-9f50-4bd2-8b0e-0044278d820d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pv6t\" (UID: \"6e6ab204-9f50-4bd2-8b0e-0044278d820d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"
Oct 30 04:02:58.552604 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:58.552564    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6e6ab204-9f50-4bd2-8b0e-0044278d820d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pv6t\" (UID: \"6e6ab204-9f50-4bd2-8b0e-0044278d820d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"
Oct 30 04:02:58.552777 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:58.552610    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6e6ab204-9f50-4bd2-8b0e-0044278d820d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pv6t\" (UID: \"6e6ab204-9f50-4bd2-8b0e-0044278d820d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"
Oct 30 04:02:58.552777 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:58.552635    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e6ab204-9f50-4bd2-8b0e-0044278d820d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pv6t\" (UID: \"6e6ab204-9f50-4bd2-8b0e-0044278d820d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"
Oct 30 04:02:58.552777 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:58.552664    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e6ab204-9f50-4bd2-8b0e-0044278d820d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pv6t\" (UID: \"6e6ab204-9f50-4bd2-8b0e-0044278d820d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"
Oct 30 04:02:58.552777 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:58.552681    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e6ab204-9f50-4bd2-8b0e-0044278d820d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pv6t\" (UID: \"6e6ab204-9f50-4bd2-8b0e-0044278d820d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"
Oct 30 04:02:58.552982 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:58.552781    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf2tn\" (UniqueName: \"kubernetes.io/projected/6e6ab204-9f50-4bd2-8b0e-0044278d820d-kube-api-access-pf2tn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pv6t\" (UID: \"6e6ab204-9f50-4bd2-8b0e-0044278d820d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"
Oct 30 04:02:58.552982 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:58.552847    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6e6ab204-9f50-4bd2-8b0e-0044278d820d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pv6t\" (UID: \"6e6ab204-9f50-4bd2-8b0e-0044278d820d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"
Oct 30 04:02:58.553398 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:58.553370    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6e6ab204-9f50-4bd2-8b0e-0044278d820d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pv6t\" (UID: \"6e6ab204-9f50-4bd2-8b0e-0044278d820d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"
Oct 30 04:02:58.555100 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:58.555072    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e6ab204-9f50-4bd2-8b0e-0044278d820d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pv6t\" (UID: \"6e6ab204-9f50-4bd2-8b0e-0044278d820d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"
Oct 30 04:02:58.555210 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:58.555113    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6e6ab204-9f50-4bd2-8b0e-0044278d820d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pv6t\" (UID: \"6e6ab204-9f50-4bd2-8b0e-0044278d820d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"
Oct 30 04:02:58.555414 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:58.555391    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e6ab204-9f50-4bd2-8b0e-0044278d820d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pv6t\" (UID: \"6e6ab204-9f50-4bd2-8b0e-0044278d820d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"
Oct 30 04:02:58.555454 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:58.555430    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6e6ab204-9f50-4bd2-8b0e-0044278d820d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pv6t\" (UID: \"6e6ab204-9f50-4bd2-8b0e-0044278d820d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"
Oct 30 04:02:58.574700 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:58.574674    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e6ab204-9f50-4bd2-8b0e-0044278d820d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pv6t\" (UID: \"6e6ab204-9f50-4bd2-8b0e-0044278d820d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"
Oct 30 04:02:58.574954 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:58.574936    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf2tn\" (UniqueName: \"kubernetes.io/projected/6e6ab204-9f50-4bd2-8b0e-0044278d820d-kube-api-access-pf2tn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7pv6t\" (UID: \"6e6ab204-9f50-4bd2-8b0e-0044278d820d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"
Oct 30 04:02:58.719817 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:58.719746    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"
Oct 30 04:02:59.044727 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:59.044692    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"]
Oct 30 04:02:59.047796 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:02:59.047767    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e6ab204_9f50_4bd2_8b0e_0044278d820d.slice/crio-29ec99d73518b72cd95e26961a80066cfe516071b557feedb9555b6a02342e98 WatchSource:0}: Error finding container 29ec99d73518b72cd95e26961a80066cfe516071b557feedb9555b6a02342e98: Status 404 returned error can't find the container with id 29ec99d73518b72cd95e26961a80066cfe516071b557feedb9555b6a02342e98
Oct 30 04:02:59.049910 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:59.049882    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:02:59.049968 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:59.049943    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:02:59.775402 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:59.775361    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t" event={"ID":"6e6ab204-9f50-4bd2-8b0e-0044278d820d","Type":"ContainerStarted","Data":"4a842fce6557c0433c1aa4adab2f73e938d8164143ed6300e6c1da1bf4f1a128"}
Oct 30 04:02:59.775798 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:59.775420    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t" event={"ID":"6e6ab204-9f50-4bd2-8b0e-0044278d820d","Type":"ContainerStarted","Data":"29ec99d73518b72cd95e26961a80066cfe516071b557feedb9555b6a02342e98"}
Oct 30 04:02:59.775798 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:59.775446    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"
Oct 30 04:02:59.834576 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:02:59.834526    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t" podStartSLOduration=1.83451202 podStartE2EDuration="1.83451202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:02:59.833101569 +0000 UTC m=+299.245497142" watchObservedRunningTime="2026-10-30 04:02:59.83451202 +0000 UTC m=+299.246907594"
Oct 30 04:03:00.741006 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:00.740979    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 04:03:00.741756 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:00.741738    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 04:03:00.780539 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:00.780516    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7pv6t"
Oct 30 04:03:00.867354 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:00.867320    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"]
Oct 30 04:03:00.868897 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:00.868829    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp" podUID="8ee8ebc1-e44c-447c-95b9-566ee210c2c1" containerName="discovery" containerID="cri-o://d7e053cfa77bd4fbdee90500dd99e39400e8ae4a0d84620c031db0f5189c3ba3" gracePeriod=30
Oct 30 04:03:01.111957 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.111934    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:03:01.170741 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.170704    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-csr-ca-configmap\") pod \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") "
Oct 30 04:03:01.170882 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.170820    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-csr-dns-cert\") pod \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") "
Oct 30 04:03:01.170882 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.170858    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-local-certs\") pod \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") "
Oct 30 04:03:01.171000 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.170886    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-cacerts\" (UniqueName: \"kubernetes.io/secret/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-cacerts\") pod \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") "
Oct 30 04:03:01.171054 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.171024    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-kube-api-access-4ztvw\" (UniqueName: \"kubernetes.io/projected/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-kube-api-access-4ztvw\") pod \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") "
Oct 30 04:03:01.171105 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.171092    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-kubeconfig\") pod \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") "
Oct 30 04:03:01.171162 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.171117    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-csr-ca-configmap" pod "8ee8ebc1-e44c-447c-95b9-566ee210c2c1" (UID: "8ee8ebc1-e44c-447c-95b9-566ee210c2c1"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:03:01.171162 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.171148    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-token\" (UniqueName: \"kubernetes.io/projected/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-token\") pod \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\" (UID: \"8ee8ebc1-e44c-447c-95b9-566ee210c2c1\") "
Oct 30 04:03:01.171430 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.171400    4874 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-csr-ca-configmap\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:03:01.173387 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.173347    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-kube-api-access-4ztvw" pod "8ee8ebc1-e44c-447c-95b9-566ee210c2c1" (UID: "8ee8ebc1-e44c-447c-95b9-566ee210c2c1"). InnerVolumeSpecName "kube-api-access-4ztvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:03:01.173570 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.173544    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-local-certs" pod "8ee8ebc1-e44c-447c-95b9-566ee210c2c1" (UID: "8ee8ebc1-e44c-447c-95b9-566ee210c2c1"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:01.173645 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.173612    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-csr-dns-cert" pod "8ee8ebc1-e44c-447c-95b9-566ee210c2c1" (UID: "8ee8ebc1-e44c-447c-95b9-566ee210c2c1"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:03:01.173710 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.173622    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-kubeconfig" pod "8ee8ebc1-e44c-447c-95b9-566ee210c2c1" (UID: "8ee8ebc1-e44c-447c-95b9-566ee210c2c1"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:03:01.173710 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.173681    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-token" pod "8ee8ebc1-e44c-447c-95b9-566ee210c2c1" (UID: "8ee8ebc1-e44c-447c-95b9-566ee210c2c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:03:01.173804 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.173780    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-cacerts" pod "8ee8ebc1-e44c-447c-95b9-566ee210c2c1" (UID: "8ee8ebc1-e44c-447c-95b9-566ee210c2c1"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:03:01.271938 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.271871    4874 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-kubeconfig\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:03:01.271938 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.271898    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:03:01.271938 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.271907    4874 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-istio-csr-dns-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:03:01.271938 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.271921    4874 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-local-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:03:01.271938 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.271941    4874 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-cacerts\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:03:01.272182 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.271954    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ztvw\" (UniqueName: \"kubernetes.io/projected/8ee8ebc1-e44c-447c-95b9-566ee210c2c1-kube-api-access-4ztvw\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:03:01.783971 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.783940    4874 generic.go:356] "Generic (PLEG): container finished" podID="8ee8ebc1-e44c-447c-95b9-566ee210c2c1" containerID="d7e053cfa77bd4fbdee90500dd99e39400e8ae4a0d84620c031db0f5189c3ba3" exitCode=0
Oct 30 04:03:01.784482 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.783995    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"
Oct 30 04:03:01.784482 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.784029    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp" event={"ID":"8ee8ebc1-e44c-447c-95b9-566ee210c2c1","Type":"ContainerDied","Data":"d7e053cfa77bd4fbdee90500dd99e39400e8ae4a0d84620c031db0f5189c3ba3"}
Oct 30 04:03:01.784482 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.784082    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp" event={"ID":"8ee8ebc1-e44c-447c-95b9-566ee210c2c1","Type":"ContainerDied","Data":"5bf1326fe0e925f3b4758186edde1a30c98487bd6de9a804ca74beaaf7eab61a"}
Oct 30 04:03:01.784482 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.784103    4874 scope.go:122] "RemoveContainer" containerID="d7e053cfa77bd4fbdee90500dd99e39400e8ae4a0d84620c031db0f5189c3ba3"
Oct 30 04:03:01.792881 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.792866    4874 scope.go:122] "RemoveContainer" containerID="d7e053cfa77bd4fbdee90500dd99e39400e8ae4a0d84620c031db0f5189c3ba3"
Oct 30 04:03:01.793141 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:03:01.793122    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7e053cfa77bd4fbdee90500dd99e39400e8ae4a0d84620c031db0f5189c3ba3\": container with ID starting with d7e053cfa77bd4fbdee90500dd99e39400e8ae4a0d84620c031db0f5189c3ba3 not found: ID does not exist" containerID="d7e053cfa77bd4fbdee90500dd99e39400e8ae4a0d84620c031db0f5189c3ba3"
Oct 30 04:03:01.793194 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.793151    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e053cfa77bd4fbdee90500dd99e39400e8ae4a0d84620c031db0f5189c3ba3"} err="failed to get container status \"d7e053cfa77bd4fbdee90500dd99e39400e8ae4a0d84620c031db0f5189c3ba3\": rpc error: code = NotFound desc = could not find container \"d7e053cfa77bd4fbdee90500dd99e39400e8ae4a0d84620c031db0f5189c3ba3\": container with ID starting with d7e053cfa77bd4fbdee90500dd99e39400e8ae4a0d84620c031db0f5189c3ba3 not found: ID does not exist"
Oct 30 04:03:01.834496 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.834464    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"]
Oct 30 04:03:01.845778 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:01.845752    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-wg2xp"]
Oct 30 04:03:02.726193 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:03:02.726158    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ee8ebc1-e44c-447c-95b9-566ee210c2c1" path="/var/lib/kubelet/pods/8ee8ebc1-e44c-447c-95b9-566ee210c2c1/volumes"
Oct 30 04:03:25.607942 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:03:25.607859    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:03:55.607556 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:03:55.607515    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:04:25.608114 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:04:25.608079    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:04:55.611772 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:04:55.611738    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:05:25.607984 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:05:25.607947    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:05:55.606770 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:05:55.606733    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:06:25.608073 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:06:25.607993    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:06:55.607995 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:06:55.607957    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:07:25.609436 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:07:25.609325    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:07:55.607689 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:07:55.607653    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:08:00.766903 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:08:00.766870    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 04:08:00.768003 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:08:00.767988    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 04:08:25.609142 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:08:25.609099    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:08:55.607904 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:08:55.607869    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:09:25.607902 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:09:25.607822    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:09:55.607330 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:09:55.607293    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:10:25.608104 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:10:25.608058    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:10:55.607266 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:10:55.607216    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:11:25.607557 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:11:25.607522    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:11:55.608025 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:11:55.607986    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:12:25.607782 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:12:25.607746    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:12:38.051829 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:38.051787    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-k9mm7"]
Oct 30 04:12:38.052407 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:38.052169    4874 cpu_manager.go:414] "containerMap: removing container" podUID="8ee8ebc1-e44c-447c-95b9-566ee210c2c1" containerName="discovery"
Oct 30 04:12:38.052407 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:38.052187    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee8ebc1-e44c-447c-95b9-566ee210c2c1" containerName="discovery"
Oct 30 04:12:38.052407 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:38.052283    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee8ebc1-e44c-447c-95b9-566ee210c2c1" containerName="discovery"
Oct 30 04:12:38.054103 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:38.054081    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-k9mm7"
Oct 30 04:12:38.072853 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:38.072827    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 04:12:38.073971 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:38.073952    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-k9mm7"]
Oct 30 04:12:38.146038 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:38.146000    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c84x\" (UniqueName: \"kubernetes.io/projected/e0ad1041-a26e-40de-99c1-8815ad14f58b-kube-api-access-7c84x\") pod \"limitador-limitador-74dfcfb894-k9mm7\" (UID: \"e0ad1041-a26e-40de-99c1-8815ad14f58b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-k9mm7"
Oct 30 04:12:38.146190 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:38.146119    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e0ad1041-a26e-40de-99c1-8815ad14f58b-config-file\") pod \"limitador-limitador-74dfcfb894-k9mm7\" (UID: \"e0ad1041-a26e-40de-99c1-8815ad14f58b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-k9mm7"
Oct 30 04:12:38.193719 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:38.193688    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-k9mm7"]
Oct 30 04:12:38.247071 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:38.247042    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e0ad1041-a26e-40de-99c1-8815ad14f58b-config-file\") pod \"limitador-limitador-74dfcfb894-k9mm7\" (UID: \"e0ad1041-a26e-40de-99c1-8815ad14f58b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-k9mm7"
Oct 30 04:12:38.247213 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:38.247091    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c84x\" (UniqueName: \"kubernetes.io/projected/e0ad1041-a26e-40de-99c1-8815ad14f58b-kube-api-access-7c84x\") pod \"limitador-limitador-74dfcfb894-k9mm7\" (UID: \"e0ad1041-a26e-40de-99c1-8815ad14f58b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-k9mm7"
Oct 30 04:12:38.247648 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:38.247626    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e0ad1041-a26e-40de-99c1-8815ad14f58b-config-file\") pod \"limitador-limitador-74dfcfb894-k9mm7\" (UID: \"e0ad1041-a26e-40de-99c1-8815ad14f58b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-k9mm7"
Oct 30 04:12:38.270086 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:38.270055    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c84x\" (UniqueName: \"kubernetes.io/projected/e0ad1041-a26e-40de-99c1-8815ad14f58b-kube-api-access-7c84x\") pod \"limitador-limitador-74dfcfb894-k9mm7\" (UID: \"e0ad1041-a26e-40de-99c1-8815ad14f58b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-k9mm7"
Oct 30 04:12:38.365164 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:38.365105    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-k9mm7"
Oct 30 04:12:38.658266 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:38.658087    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-k9mm7"]
Oct 30 04:12:38.660490 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:12:38.660464    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ad1041_a26e_40de_99c1_8815ad14f58b.slice/crio-736fbfa0a909b11dc052b2b7a0f16bb87670c8a8dd1019ca371f6bf996bc347f WatchSource:0}: Error finding container 736fbfa0a909b11dc052b2b7a0f16bb87670c8a8dd1019ca371f6bf996bc347f: Status 404 returned error can't find the container with id 736fbfa0a909b11dc052b2b7a0f16bb87670c8a8dd1019ca371f6bf996bc347f
Oct 30 04:12:38.662826 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:38.662806    4874 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 04:12:38.759197 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:38.759168    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-k9mm7" event={"ID":"e0ad1041-a26e-40de-99c1-8815ad14f58b","Type":"ContainerStarted","Data":"736fbfa0a909b11dc052b2b7a0f16bb87670c8a8dd1019ca371f6bf996bc347f"}
Oct 30 04:12:38.910650 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:38.910563    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pnhp4"]
Oct 30 04:12:38.913601 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:38.913578    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pnhp4"
Oct 30 04:12:38.930803 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:38.930782    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pnhp4"]
Oct 30 04:12:38.936186 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:38.936166    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rzxvm\""
Oct 30 04:12:38.952014 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:38.951990    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwbt8\" (UniqueName: \"kubernetes.io/projected/96f44c93-dbf4-45a0-be39-bfb0e4c9ee60-kube-api-access-fwbt8\") pod \"authorino-57f7f8659f-pnhp4\" (UID: \"96f44c93-dbf4-45a0-be39-bfb0e4c9ee60\") " pod="kuadrant-system/authorino-57f7f8659f-pnhp4"
Oct 30 04:12:39.053395 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:39.053365    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwbt8\" (UniqueName: \"kubernetes.io/projected/96f44c93-dbf4-45a0-be39-bfb0e4c9ee60-kube-api-access-fwbt8\") pod \"authorino-57f7f8659f-pnhp4\" (UID: \"96f44c93-dbf4-45a0-be39-bfb0e4c9ee60\") " pod="kuadrant-system/authorino-57f7f8659f-pnhp4"
Oct 30 04:12:39.075794 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:39.075766    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwbt8\" (UniqueName: \"kubernetes.io/projected/96f44c93-dbf4-45a0-be39-bfb0e4c9ee60-kube-api-access-fwbt8\") pod \"authorino-57f7f8659f-pnhp4\" (UID: \"96f44c93-dbf4-45a0-be39-bfb0e4c9ee60\") " pod="kuadrant-system/authorino-57f7f8659f-pnhp4"
Oct 30 04:12:39.223426 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:39.223364    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pnhp4"
Oct 30 04:12:39.518944 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:39.518917    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pnhp4"]
Oct 30 04:12:39.521193 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:12:39.521162    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96f44c93_dbf4_45a0_be39_bfb0e4c9ee60.slice/crio-8f0f0bde86517d57943b4a87f233d39c12388d75ab43a2e5f1176e6d6c112ace WatchSource:0}: Error finding container 8f0f0bde86517d57943b4a87f233d39c12388d75ab43a2e5f1176e6d6c112ace: Status 404 returned error can't find the container with id 8f0f0bde86517d57943b4a87f233d39c12388d75ab43a2e5f1176e6d6c112ace
Oct 30 04:12:39.595111 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:39.595076    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:12:39.597745 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:39.597715    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-vg5td"
Oct 30 04:12:39.620744 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:39.620717    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:12:39.658767 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:39.658734    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm282\" (UniqueName: \"kubernetes.io/projected/b90621d3-240a-489c-9ed2-6c1913a13972-kube-api-access-tm282\") pod \"limitador-limitador-7c9b87d574-vg5td\" (UID: \"b90621d3-240a-489c-9ed2-6c1913a13972\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vg5td"
Oct 30 04:12:39.658914 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:39.658808    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b90621d3-240a-489c-9ed2-6c1913a13972-config-file\") pod \"limitador-limitador-7c9b87d574-vg5td\" (UID: \"b90621d3-240a-489c-9ed2-6c1913a13972\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vg5td"
Oct 30 04:12:39.691969 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:39.691937    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:12:39.760179 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:39.760138    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm282\" (UniqueName: \"kubernetes.io/projected/b90621d3-240a-489c-9ed2-6c1913a13972-kube-api-access-tm282\") pod \"limitador-limitador-7c9b87d574-vg5td\" (UID: \"b90621d3-240a-489c-9ed2-6c1913a13972\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vg5td"
Oct 30 04:12:39.760359 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:39.760221    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b90621d3-240a-489c-9ed2-6c1913a13972-config-file\") pod \"limitador-limitador-7c9b87d574-vg5td\" (UID: \"b90621d3-240a-489c-9ed2-6c1913a13972\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vg5td"
Oct 30 04:12:39.761061 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:39.761032    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b90621d3-240a-489c-9ed2-6c1913a13972-config-file\") pod \"limitador-limitador-7c9b87d574-vg5td\" (UID: \"b90621d3-240a-489c-9ed2-6c1913a13972\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vg5td"
Oct 30 04:12:39.766231 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:39.766203    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pnhp4" event={"ID":"96f44c93-dbf4-45a0-be39-bfb0e4c9ee60","Type":"ContainerStarted","Data":"8f0f0bde86517d57943b4a87f233d39c12388d75ab43a2e5f1176e6d6c112ace"}
Oct 30 04:12:39.782819 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:39.782766    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm282\" (UniqueName: \"kubernetes.io/projected/b90621d3-240a-489c-9ed2-6c1913a13972-kube-api-access-tm282\") pod \"limitador-limitador-7c9b87d574-vg5td\" (UID: \"b90621d3-240a-489c-9ed2-6c1913a13972\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vg5td"
Oct 30 04:12:39.913968 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:39.913922    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-vg5td"
Oct 30 04:12:40.212005 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:40.211892    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:12:40.217623 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:12:40.216536    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb90621d3_240a_489c_9ed2_6c1913a13972.slice/crio-8fef28179d09465a68b3836153333a70e4744fffdb90a2eb01261520419c6ec1 WatchSource:0}: Error finding container 8fef28179d09465a68b3836153333a70e4744fffdb90a2eb01261520419c6ec1: Status 404 returned error can't find the container with id 8fef28179d09465a68b3836153333a70e4744fffdb90a2eb01261520419c6ec1
Oct 30 04:12:40.774824 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:40.774741    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-vg5td" event={"ID":"b90621d3-240a-489c-9ed2-6c1913a13972","Type":"ContainerStarted","Data":"8fef28179d09465a68b3836153333a70e4744fffdb90a2eb01261520419c6ec1"}
Oct 30 04:12:41.780421 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:41.780388    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-vg5td" event={"ID":"b90621d3-240a-489c-9ed2-6c1913a13972","Type":"ContainerStarted","Data":"421929255d18ffac0b23aa9348d7dc6344f85b3af528e1ddde66adbf0bc549e2"}
Oct 30 04:12:41.780907 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:41.780478    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-vg5td"
Oct 30 04:12:41.781826 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:41.781804    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-k9mm7" event={"ID":"e0ad1041-a26e-40de-99c1-8815ad14f58b","Type":"ContainerStarted","Data":"da9a44d25bb25eaaeee6eed28662a3c722d248b88f31b1164ca84b96eba63799"}
Oct 30 04:12:41.781952 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:41.781937    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-k9mm7"
Oct 30 04:12:41.819158 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:41.819079    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-vg5td" podStartSLOduration=1.465029498 podStartE2EDuration="2.819068848s" totalImagesPullingTime="1.35403935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:12:41.81859032 +0000 UTC m=+881.230985892" watchObservedRunningTime="2026-10-30 04:12:41.819068848 +0000 UTC m=+881.231464421"
Oct 30 04:12:41.860296 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:41.860218    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-k9mm7" podStartSLOduration=0.957588261 podStartE2EDuration="3.860204512s" totalImagesPullingTime="2.902616251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:12:41.859775556 +0000 UTC m=+881.272171131" watchObservedRunningTime="2026-10-30 04:12:41.860204512 +0000 UTC m=+881.272600105"
Oct 30 04:12:41.872719 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:41.872660    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pnhp4"]
Oct 30 04:12:42.786300 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:42.786265    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pnhp4" event={"ID":"96f44c93-dbf4-45a0-be39-bfb0e4c9ee60","Type":"ContainerStarted","Data":"60f158e8895a8439a9823b33d072a73742275eeedfb7f9e70556f8940d4dd2c2"}
Oct 30 04:12:42.786300 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:42.786274    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-pnhp4" podUID="96f44c93-dbf4-45a0-be39-bfb0e4c9ee60" containerName="authorino" containerID="cri-o://60f158e8895a8439a9823b33d072a73742275eeedfb7f9e70556f8940d4dd2c2" gracePeriod=30
Oct 30 04:12:42.796131 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:12:42.796103    4874 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96f44c93_dbf4_45a0_be39_bfb0e4c9ee60.slice/crio-60f158e8895a8439a9823b33d072a73742275eeedfb7f9e70556f8940d4dd2c2.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 04:12:42.822633 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:42.822592    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-pnhp4" podStartSLOduration=2.059587951 podStartE2EDuration="4.822580765s" totalImagesPullingTime="2.762992814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:12:42.822094151 +0000 UTC m=+882.234489724" watchObservedRunningTime="2026-10-30 04:12:42.822580765 +0000 UTC m=+882.234976354"
Oct 30 04:12:43.020031 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:43.020009    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pnhp4"
Oct 30 04:12:43.089516 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:43.089444    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96f44c93-dbf4-45a0-be39-bfb0e4c9ee60-kube-api-access-fwbt8\" (UniqueName: \"kubernetes.io/projected/96f44c93-dbf4-45a0-be39-bfb0e4c9ee60-kube-api-access-fwbt8\") pod \"96f44c93-dbf4-45a0-be39-bfb0e4c9ee60\" (UID: \"96f44c93-dbf4-45a0-be39-bfb0e4c9ee60\") "
Oct 30 04:12:43.091621 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:43.091587    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f44c93-dbf4-45a0-be39-bfb0e4c9ee60-kube-api-access-fwbt8" pod "96f44c93-dbf4-45a0-be39-bfb0e4c9ee60" (UID: "96f44c93-dbf4-45a0-be39-bfb0e4c9ee60"). InnerVolumeSpecName "kube-api-access-fwbt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:43.190365 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:43.190334    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwbt8\" (UniqueName: \"kubernetes.io/projected/96f44c93-dbf4-45a0-be39-bfb0e4c9ee60-kube-api-access-fwbt8\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:12:43.791451 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:43.791414    4874 generic.go:356] "Generic (PLEG): container finished" podID="96f44c93-dbf4-45a0-be39-bfb0e4c9ee60" containerID="60f158e8895a8439a9823b33d072a73742275eeedfb7f9e70556f8940d4dd2c2" exitCode=0
Oct 30 04:12:43.791924 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:43.791459    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pnhp4"
Oct 30 04:12:43.791924 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:43.791507    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pnhp4" event={"ID":"96f44c93-dbf4-45a0-be39-bfb0e4c9ee60","Type":"ContainerDied","Data":"60f158e8895a8439a9823b33d072a73742275eeedfb7f9e70556f8940d4dd2c2"}
Oct 30 04:12:43.791924 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:43.791577    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pnhp4" event={"ID":"96f44c93-dbf4-45a0-be39-bfb0e4c9ee60","Type":"ContainerDied","Data":"8f0f0bde86517d57943b4a87f233d39c12388d75ab43a2e5f1176e6d6c112ace"}
Oct 30 04:12:43.791924 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:43.791603    4874 scope.go:122] "RemoveContainer" containerID="60f158e8895a8439a9823b33d072a73742275eeedfb7f9e70556f8940d4dd2c2"
Oct 30 04:12:43.802086 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:43.801783    4874 scope.go:122] "RemoveContainer" containerID="60f158e8895a8439a9823b33d072a73742275eeedfb7f9e70556f8940d4dd2c2"
Oct 30 04:12:43.802174 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:12:43.802077    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f158e8895a8439a9823b33d072a73742275eeedfb7f9e70556f8940d4dd2c2\": container with ID starting with 60f158e8895a8439a9823b33d072a73742275eeedfb7f9e70556f8940d4dd2c2 not found: ID does not exist" containerID="60f158e8895a8439a9823b33d072a73742275eeedfb7f9e70556f8940d4dd2c2"
Oct 30 04:12:43.802174 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:43.802110    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f158e8895a8439a9823b33d072a73742275eeedfb7f9e70556f8940d4dd2c2"} err="failed to get container status \"60f158e8895a8439a9823b33d072a73742275eeedfb7f9e70556f8940d4dd2c2\": rpc error: code = NotFound desc = could not find container \"60f158e8895a8439a9823b33d072a73742275eeedfb7f9e70556f8940d4dd2c2\": container with ID starting with 60f158e8895a8439a9823b33d072a73742275eeedfb7f9e70556f8940d4dd2c2 not found: ID does not exist"
Oct 30 04:12:43.839507 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:43.839485    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pnhp4"]
Oct 30 04:12:43.851006 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:43.850975    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pnhp4"]
Oct 30 04:12:44.726356 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:44.726325    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96f44c93-dbf4-45a0-be39-bfb0e4c9ee60" path="/var/lib/kubelet/pods/96f44c93-dbf4-45a0-be39-bfb0e4c9ee60/volumes"
Oct 30 04:12:52.787760 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:52.787720    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-k9mm7"
Oct 30 04:12:52.788159 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:52.787795    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-vg5td"
Oct 30 04:12:52.912499 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:52.912469    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-k9mm7"]
Oct 30 04:12:52.912651 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:52.912614    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-k9mm7" podUID="e0ad1041-a26e-40de-99c1-8815ad14f58b" containerName="limitador" containerID="cri-o://da9a44d25bb25eaaeee6eed28662a3c722d248b88f31b1164ca84b96eba63799" gracePeriod=30
Oct 30 04:12:53.456280 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:53.456256    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-k9mm7"
Oct 30 04:12:53.574953 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:53.574927    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0ad1041-a26e-40de-99c1-8815ad14f58b-config-file\" (UniqueName: \"kubernetes.io/configmap/e0ad1041-a26e-40de-99c1-8815ad14f58b-config-file\") pod \"e0ad1041-a26e-40de-99c1-8815ad14f58b\" (UID: \"e0ad1041-a26e-40de-99c1-8815ad14f58b\") "
Oct 30 04:12:53.575113 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:53.574973    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ad1041-a26e-40de-99c1-8815ad14f58b-kube-api-access-7c84x\" (UniqueName: \"kubernetes.io/projected/e0ad1041-a26e-40de-99c1-8815ad14f58b-kube-api-access-7c84x\") pod \"e0ad1041-a26e-40de-99c1-8815ad14f58b\" (UID: \"e0ad1041-a26e-40de-99c1-8815ad14f58b\") "
Oct 30 04:12:53.575360 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:53.575334    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0ad1041-a26e-40de-99c1-8815ad14f58b-config-file" pod "e0ad1041-a26e-40de-99c1-8815ad14f58b" (UID: "e0ad1041-a26e-40de-99c1-8815ad14f58b"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:53.576997 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:53.576973    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ad1041-a26e-40de-99c1-8815ad14f58b-kube-api-access-7c84x" pod "e0ad1041-a26e-40de-99c1-8815ad14f58b" (UID: "e0ad1041-a26e-40de-99c1-8815ad14f58b"). InnerVolumeSpecName "kube-api-access-7c84x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:53.676317 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:53.676283    4874 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e0ad1041-a26e-40de-99c1-8815ad14f58b-config-file\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:12:53.676317 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:53.676311    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c84x\" (UniqueName: \"kubernetes.io/projected/e0ad1041-a26e-40de-99c1-8815ad14f58b-kube-api-access-7c84x\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:12:53.832529 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:53.832432    4874 generic.go:356] "Generic (PLEG): container finished" podID="e0ad1041-a26e-40de-99c1-8815ad14f58b" containerID="da9a44d25bb25eaaeee6eed28662a3c722d248b88f31b1164ca84b96eba63799" exitCode=0
Oct 30 04:12:53.832529 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:53.832478    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-k9mm7"
Oct 30 04:12:53.832529 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:53.832497    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-k9mm7" event={"ID":"e0ad1041-a26e-40de-99c1-8815ad14f58b","Type":"ContainerDied","Data":"da9a44d25bb25eaaeee6eed28662a3c722d248b88f31b1164ca84b96eba63799"}
Oct 30 04:12:53.833011 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:53.832551    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-k9mm7" event={"ID":"e0ad1041-a26e-40de-99c1-8815ad14f58b","Type":"ContainerDied","Data":"736fbfa0a909b11dc052b2b7a0f16bb87670c8a8dd1019ca371f6bf996bc347f"}
Oct 30 04:12:53.833011 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:53.832582    4874 scope.go:122] "RemoveContainer" containerID="da9a44d25bb25eaaeee6eed28662a3c722d248b88f31b1164ca84b96eba63799"
Oct 30 04:12:53.840580 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:53.840559    4874 scope.go:122] "RemoveContainer" containerID="da9a44d25bb25eaaeee6eed28662a3c722d248b88f31b1164ca84b96eba63799"
Oct 30 04:12:53.840853 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:12:53.840835    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da9a44d25bb25eaaeee6eed28662a3c722d248b88f31b1164ca84b96eba63799\": container with ID starting with da9a44d25bb25eaaeee6eed28662a3c722d248b88f31b1164ca84b96eba63799 not found: ID does not exist" containerID="da9a44d25bb25eaaeee6eed28662a3c722d248b88f31b1164ca84b96eba63799"
Oct 30 04:12:53.840903 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:53.840861    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da9a44d25bb25eaaeee6eed28662a3c722d248b88f31b1164ca84b96eba63799"} err="failed to get container status \"da9a44d25bb25eaaeee6eed28662a3c722d248b88f31b1164ca84b96eba63799\": rpc error: code = NotFound desc = could not find container \"da9a44d25bb25eaaeee6eed28662a3c722d248b88f31b1164ca84b96eba63799\": container with ID starting with da9a44d25bb25eaaeee6eed28662a3c722d248b88f31b1164ca84b96eba63799 not found: ID does not exist"
Oct 30 04:12:53.882590 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:53.882564    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-k9mm7"]
Oct 30 04:12:53.893919 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:53.893901    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-k9mm7"]
Oct 30 04:12:54.727076 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:12:54.727047    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0ad1041-a26e-40de-99c1-8815ad14f58b" path="/var/lib/kubelet/pods/e0ad1041-a26e-40de-99c1-8815ad14f58b/volumes"
Oct 30 04:12:55.607677 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:12:55.607641    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:13:00.792347 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:00.792324    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 04:13:00.802418 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:00.802397    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 04:13:10.889041 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:10.889006    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-22p96"]
Oct 30 04:13:10.889543 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:10.889386    4874 cpu_manager.go:414] "containerMap: removing container" podUID="e0ad1041-a26e-40de-99c1-8815ad14f58b" containerName="limitador"
Oct 30 04:13:10.889543 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:10.889404    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ad1041-a26e-40de-99c1-8815ad14f58b" containerName="limitador"
Oct 30 04:13:10.889543 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:10.889424    4874 cpu_manager.go:414] "containerMap: removing container" podUID="96f44c93-dbf4-45a0-be39-bfb0e4c9ee60" containerName="authorino"
Oct 30 04:13:10.889543 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:10.889430    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96f44c93-dbf4-45a0-be39-bfb0e4c9ee60" containerName="authorino"
Oct 30 04:13:10.889543 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:10.889481    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ad1041-a26e-40de-99c1-8815ad14f58b" containerName="limitador"
Oct 30 04:13:10.889543 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:10.889491    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="96f44c93-dbf4-45a0-be39-bfb0e4c9ee60" containerName="authorino"
Oct 30 04:13:10.893771 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:10.893752    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-22p96"
Oct 30 04:13:10.921331 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:10.921306    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 04:13:10.921484 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:10.921357    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 04:13:10.921560 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:10.921444    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 30 04:13:10.921560 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:10.921528    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-mdcjz\""
Oct 30 04:13:10.922743 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:10.922719    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-22p96"]
Oct 30 04:13:10.993749 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:10.993726    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd8p8\" (UniqueName: \"kubernetes.io/projected/5acd9635-8003-4366-b5ed-a5ad9641d38e-kube-api-access-vd8p8\") pod \"postgres-7cdbf6dc44-22p96\" (UID: \"5acd9635-8003-4366-b5ed-a5ad9641d38e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-22p96"
Oct 30 04:13:10.993894 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:10.993770    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5acd9635-8003-4366-b5ed-a5ad9641d38e-data\") pod \"postgres-7cdbf6dc44-22p96\" (UID: \"5acd9635-8003-4366-b5ed-a5ad9641d38e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-22p96"
Oct 30 04:13:11.094625 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:11.094596    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd8p8\" (UniqueName: \"kubernetes.io/projected/5acd9635-8003-4366-b5ed-a5ad9641d38e-kube-api-access-vd8p8\") pod \"postgres-7cdbf6dc44-22p96\" (UID: \"5acd9635-8003-4366-b5ed-a5ad9641d38e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-22p96"
Oct 30 04:13:11.094768 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:11.094637    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5acd9635-8003-4366-b5ed-a5ad9641d38e-data\") pod \"postgres-7cdbf6dc44-22p96\" (UID: \"5acd9635-8003-4366-b5ed-a5ad9641d38e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-22p96"
Oct 30 04:13:11.094983 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:11.094967    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5acd9635-8003-4366-b5ed-a5ad9641d38e-data\") pod \"postgres-7cdbf6dc44-22p96\" (UID: \"5acd9635-8003-4366-b5ed-a5ad9641d38e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-22p96"
Oct 30 04:13:11.117750 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:11.117724    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd8p8\" (UniqueName: \"kubernetes.io/projected/5acd9635-8003-4366-b5ed-a5ad9641d38e-kube-api-access-vd8p8\") pod \"postgres-7cdbf6dc44-22p96\" (UID: \"5acd9635-8003-4366-b5ed-a5ad9641d38e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-22p96"
Oct 30 04:13:11.205990 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:11.205925    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-22p96"
Oct 30 04:13:11.497365 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:11.497344    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-22p96"]
Oct 30 04:13:11.499218 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:13:11.499192    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5acd9635_8003_4366_b5ed_a5ad9641d38e.slice/crio-caa2531cc700e406a2b18226927c2a140881374ab1fb6114ac373dae0281cf09 WatchSource:0}: Error finding container caa2531cc700e406a2b18226927c2a140881374ab1fb6114ac373dae0281cf09: Status 404 returned error can't find the container with id caa2531cc700e406a2b18226927c2a140881374ab1fb6114ac373dae0281cf09
Oct 30 04:13:11.899521 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:11.899442    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-22p96" event={"ID":"5acd9635-8003-4366-b5ed-a5ad9641d38e","Type":"ContainerStarted","Data":"caa2531cc700e406a2b18226927c2a140881374ab1fb6114ac373dae0281cf09"}
Oct 30 04:13:16.923298 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:16.923264    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-22p96" event={"ID":"5acd9635-8003-4366-b5ed-a5ad9641d38e","Type":"ContainerStarted","Data":"76a6a164bfce4f70194b9f6b471ea8eae9632d610a00761a38900ed03ea2bb71"}
Oct 30 04:13:16.923653 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:16.923377    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-22p96"
Oct 30 04:13:16.962044 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:16.962003    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-22p96" podStartSLOduration=2.300261231 podStartE2EDuration="6.96199187s" totalImagesPullingTime="4.661730639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:16.961574506 +0000 UTC m=+916.373970079" watchObservedRunningTime="2026-10-30 04:13:16.96199187 +0000 UTC m=+916.374387437"
Oct 30 04:13:22.956421 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:13:22.956393    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-22p96"
Oct 30 04:13:25.608152 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:13:25.608118    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:13:55.607898 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:13:55.607858    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:14:04.658763 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:04.658724    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv"]
Oct 30 04:14:04.661005 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:04.660988    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv"
Oct 30 04:14:04.686416 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:04.686394    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 04:14:04.686538 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:04.686473    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 04:14:04.686538 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:04.686487    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gw5bk\""
Oct 30 04:14:04.689428 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:04.689407    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv"]
Oct 30 04:14:04.784197 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:04.784167    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d51474-e0df-472a-a8ba-145e29d62d3c-metrics-certs\") pod \"maas-api-546d67dc88-pjqfv\" (UID: \"f1d51474-e0df-472a-a8ba-145e29d62d3c\") " pod="odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv"
Oct 30 04:14:04.784329 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:04.784209    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvcsj\" (UniqueName: \"kubernetes.io/projected/f1d51474-e0df-472a-a8ba-145e29d62d3c-kube-api-access-nvcsj\") pod \"maas-api-546d67dc88-pjqfv\" (UID: \"f1d51474-e0df-472a-a8ba-145e29d62d3c\") " pod="odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv"
Oct 30 04:14:04.784329 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:04.784310    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d51474-e0df-472a-a8ba-145e29d62d3c-openshift-service-ca\") pod \"maas-api-546d67dc88-pjqfv\" (UID: \"f1d51474-e0df-472a-a8ba-145e29d62d3c\") " pod="odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv"
Oct 30 04:14:04.784402 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:04.784362    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1d51474-e0df-472a-a8ba-145e29d62d3c-maas-api-tls\") pod \"maas-api-546d67dc88-pjqfv\" (UID: \"f1d51474-e0df-472a-a8ba-145e29d62d3c\") " pod="odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv"
Oct 30 04:14:04.885010 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:04.884985    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1d51474-e0df-472a-a8ba-145e29d62d3c-maas-api-tls\") pod \"maas-api-546d67dc88-pjqfv\" (UID: \"f1d51474-e0df-472a-a8ba-145e29d62d3c\") " pod="odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv"
Oct 30 04:14:04.885140 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:04.885015    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d51474-e0df-472a-a8ba-145e29d62d3c-metrics-certs\") pod \"maas-api-546d67dc88-pjqfv\" (UID: \"f1d51474-e0df-472a-a8ba-145e29d62d3c\") " pod="odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv"
Oct 30 04:14:04.885140 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:04.885053    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvcsj\" (UniqueName: \"kubernetes.io/projected/f1d51474-e0df-472a-a8ba-145e29d62d3c-kube-api-access-nvcsj\") pod \"maas-api-546d67dc88-pjqfv\" (UID: \"f1d51474-e0df-472a-a8ba-145e29d62d3c\") " pod="odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv"
Oct 30 04:14:04.885140 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:04.885102    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d51474-e0df-472a-a8ba-145e29d62d3c-openshift-service-ca\") pod \"maas-api-546d67dc88-pjqfv\" (UID: \"f1d51474-e0df-472a-a8ba-145e29d62d3c\") " pod="odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv"
Oct 30 04:14:04.885763 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:04.885741    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d51474-e0df-472a-a8ba-145e29d62d3c-openshift-service-ca\") pod \"maas-api-546d67dc88-pjqfv\" (UID: \"f1d51474-e0df-472a-a8ba-145e29d62d3c\") " pod="odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv"
Oct 30 04:14:04.887514 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:04.887492    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d51474-e0df-472a-a8ba-145e29d62d3c-metrics-certs\") pod \"maas-api-546d67dc88-pjqfv\" (UID: \"f1d51474-e0df-472a-a8ba-145e29d62d3c\") " pod="odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv"
Oct 30 04:14:04.887645 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:04.887626    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1d51474-e0df-472a-a8ba-145e29d62d3c-maas-api-tls\") pod \"maas-api-546d67dc88-pjqfv\" (UID: \"f1d51474-e0df-472a-a8ba-145e29d62d3c\") " pod="odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv"
Oct 30 04:14:04.907551 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:04.907511    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvcsj\" (UniqueName: \"kubernetes.io/projected/f1d51474-e0df-472a-a8ba-145e29d62d3c-kube-api-access-nvcsj\") pod \"maas-api-546d67dc88-pjqfv\" (UID: \"f1d51474-e0df-472a-a8ba-145e29d62d3c\") " pod="odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv"
Oct 30 04:14:04.972157 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:04.972091    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv"
Oct 30 04:14:05.265602 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:05.265577    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv"]
Oct 30 04:14:05.267171 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:14:05.267139    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d51474_e0df_472a_a8ba_145e29d62d3c.slice/crio-c50d852248a2c532d628505c43865ee3f8a28434585b8d105de5c6f1b1a1049d WatchSource:0}: Error finding container c50d852248a2c532d628505c43865ee3f8a28434585b8d105de5c6f1b1a1049d: Status 404 returned error can't find the container with id c50d852248a2c532d628505c43865ee3f8a28434585b8d105de5c6f1b1a1049d
Oct 30 04:14:06.111855 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:06.111817    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv" event={"ID":"f1d51474-e0df-472a-a8ba-145e29d62d3c","Type":"ContainerStarted","Data":"c50d852248a2c532d628505c43865ee3f8a28434585b8d105de5c6f1b1a1049d"}
Oct 30 04:14:06.325012 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:06.324974    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:14:08.121313 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:08.121271    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv" event={"ID":"f1d51474-e0df-472a-a8ba-145e29d62d3c","Type":"ContainerStarted","Data":"aee4c92d1b4e4dda967e09b23fcc3567e03ab98e37d0dc09c932ed926612a4d3"}
Oct 30 04:14:08.121682 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:08.121445    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv"
Oct 30 04:14:08.161747 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:08.161699    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv" podStartSLOduration=1.7685238330000002 podStartE2EDuration="4.161685913s" totalImagesPullingTime="2.39316208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:08.161058526 +0000 UTC m=+967.573454098" watchObservedRunningTime="2026-10-30 04:14:08.161685913 +0000 UTC m=+967.574081485"
Oct 30 04:14:14.130655 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:14.130628    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv"
Oct 30 04:14:15.418023 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:15.417987    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65f8dc9b64-t2xjq"]
Oct 30 04:14:15.420663 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:15.420646    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65f8dc9b64-t2xjq"
Oct 30 04:14:15.438023 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:15.437997    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65f8dc9b64-t2xjq"]
Oct 30 04:14:15.448462 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:15.448440    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 04:14:15.448558 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:15.448537    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rzxvm\""
Oct 30 04:14:15.563876 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:15.563848    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c44245be-88c9-4175-ae0b-257f5f4f425b-tls-cert\") pod \"authorino-65f8dc9b64-t2xjq\" (UID: \"c44245be-88c9-4175-ae0b-257f5f4f425b\") " pod="kuadrant-system/authorino-65f8dc9b64-t2xjq"
Oct 30 04:14:15.564014 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:15.563887    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8vxl\" (UniqueName: \"kubernetes.io/projected/c44245be-88c9-4175-ae0b-257f5f4f425b-kube-api-access-g8vxl\") pod \"authorino-65f8dc9b64-t2xjq\" (UID: \"c44245be-88c9-4175-ae0b-257f5f4f425b\") " pod="kuadrant-system/authorino-65f8dc9b64-t2xjq"
Oct 30 04:14:15.664439 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:15.664410    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c44245be-88c9-4175-ae0b-257f5f4f425b-tls-cert\") pod \"authorino-65f8dc9b64-t2xjq\" (UID: \"c44245be-88c9-4175-ae0b-257f5f4f425b\") " pod="kuadrant-system/authorino-65f8dc9b64-t2xjq"
Oct 30 04:14:15.664509 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:15.664451    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8vxl\" (UniqueName: \"kubernetes.io/projected/c44245be-88c9-4175-ae0b-257f5f4f425b-kube-api-access-g8vxl\") pod \"authorino-65f8dc9b64-t2xjq\" (UID: \"c44245be-88c9-4175-ae0b-257f5f4f425b\") " pod="kuadrant-system/authorino-65f8dc9b64-t2xjq"
Oct 30 04:14:15.666752 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:15.666727    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c44245be-88c9-4175-ae0b-257f5f4f425b-tls-cert\") pod \"authorino-65f8dc9b64-t2xjq\" (UID: \"c44245be-88c9-4175-ae0b-257f5f4f425b\") " pod="kuadrant-system/authorino-65f8dc9b64-t2xjq"
Oct 30 04:14:15.686968 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:15.686921    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8vxl\" (UniqueName: \"kubernetes.io/projected/c44245be-88c9-4175-ae0b-257f5f4f425b-kube-api-access-g8vxl\") pod \"authorino-65f8dc9b64-t2xjq\" (UID: \"c44245be-88c9-4175-ae0b-257f5f4f425b\") " pod="kuadrant-system/authorino-65f8dc9b64-t2xjq"
Oct 30 04:14:15.730887 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:15.730859    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65f8dc9b64-t2xjq"
Oct 30 04:14:16.023506 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:16.023473    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65f8dc9b64-t2xjq"]
Oct 30 04:14:16.031194 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:14:16.031163    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc44245be_88c9_4175_ae0b_257f5f4f425b.slice/crio-1a3183871815f1010e09e4afbf441e2080a0886a8e2949363b3e1c211beccf2b WatchSource:0}: Error finding container 1a3183871815f1010e09e4afbf441e2080a0886a8e2949363b3e1c211beccf2b: Status 404 returned error can't find the container with id 1a3183871815f1010e09e4afbf441e2080a0886a8e2949363b3e1c211beccf2b
Oct 30 04:14:16.151754 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:16.151718    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65f8dc9b64-t2xjq" event={"ID":"c44245be-88c9-4175-ae0b-257f5f4f425b","Type":"ContainerStarted","Data":"1a3183871815f1010e09e4afbf441e2080a0886a8e2949363b3e1c211beccf2b"}
Oct 30 04:14:17.157521 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:17.157485    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65f8dc9b64-t2xjq" event={"ID":"c44245be-88c9-4175-ae0b-257f5f4f425b","Type":"ContainerStarted","Data":"6b72236a126623b76ec524bf9e2d4a12f6ff5339f2cc93ef61ed9ad28e6d0a7e"}
Oct 30 04:14:17.195531 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:17.195474    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-65f8dc9b64-t2xjq" podStartSLOduration=1.6906156399999999 podStartE2EDuration="2.195460432s" totalImagesPullingTime="504.844792ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:17.195075434 +0000 UTC m=+976.607471007" watchObservedRunningTime="2026-10-30 04:14:17.195460432 +0000 UTC m=+976.607856005"
Oct 30 04:14:24.053789 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:24.053754    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv"]
Oct 30 04:14:24.054186 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:24.054004    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv" podUID="f1d51474-e0df-472a-a8ba-145e29d62d3c" containerName="maas-api" containerID="cri-o://aee4c92d1b4e4dda967e09b23fcc3567e03ab98e37d0dc09c932ed926612a4d3" gracePeriod=30
Oct 30 04:14:24.125356 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:24.125319    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv" podUID="f1d51474-e0df-472a-a8ba-145e29d62d3c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.23:8443/health\": dial tcp 10.134.0.23:8443: connect: connection refused"
Oct 30 04:14:24.186715 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:24.186685    4874 generic.go:356] "Generic (PLEG): container finished" podID="f1d51474-e0df-472a-a8ba-145e29d62d3c" containerID="aee4c92d1b4e4dda967e09b23fcc3567e03ab98e37d0dc09c932ed926612a4d3" exitCode=0
Oct 30 04:14:24.186855 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:24.186762    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv" event={"ID":"f1d51474-e0df-472a-a8ba-145e29d62d3c","Type":"ContainerDied","Data":"aee4c92d1b4e4dda967e09b23fcc3567e03ab98e37d0dc09c932ed926612a4d3"}
Oct 30 04:14:24.283979 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:24.283954    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv"
Oct 30 04:14:24.430995 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:24.430916    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1d51474-e0df-472a-a8ba-145e29d62d3c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d51474-e0df-472a-a8ba-145e29d62d3c-metrics-certs\") pod \"f1d51474-e0df-472a-a8ba-145e29d62d3c\" (UID: \"f1d51474-e0df-472a-a8ba-145e29d62d3c\") "
Oct 30 04:14:24.430995 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:24.430965    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d51474-e0df-472a-a8ba-145e29d62d3c-kube-api-access-nvcsj\" (UniqueName: \"kubernetes.io/projected/f1d51474-e0df-472a-a8ba-145e29d62d3c-kube-api-access-nvcsj\") pod \"f1d51474-e0df-472a-a8ba-145e29d62d3c\" (UID: \"f1d51474-e0df-472a-a8ba-145e29d62d3c\") "
Oct 30 04:14:24.431220 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:24.431085    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1d51474-e0df-472a-a8ba-145e29d62d3c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d51474-e0df-472a-a8ba-145e29d62d3c-openshift-service-ca\") pod \"f1d51474-e0df-472a-a8ba-145e29d62d3c\" (UID: \"f1d51474-e0df-472a-a8ba-145e29d62d3c\") "
Oct 30 04:14:24.431220 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:24.431127    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1d51474-e0df-472a-a8ba-145e29d62d3c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1d51474-e0df-472a-a8ba-145e29d62d3c-maas-api-tls\") pod \"f1d51474-e0df-472a-a8ba-145e29d62d3c\" (UID: \"f1d51474-e0df-472a-a8ba-145e29d62d3c\") "
Oct 30 04:14:24.431481 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:24.431451    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d51474-e0df-472a-a8ba-145e29d62d3c-openshift-service-ca" pod "f1d51474-e0df-472a-a8ba-145e29d62d3c" (UID: "f1d51474-e0df-472a-a8ba-145e29d62d3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:14:24.433205 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:24.433176    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d51474-e0df-472a-a8ba-145e29d62d3c-kube-api-access-nvcsj" pod "f1d51474-e0df-472a-a8ba-145e29d62d3c" (UID: "f1d51474-e0df-472a-a8ba-145e29d62d3c"). InnerVolumeSpecName "kube-api-access-nvcsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:14:24.433205 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:24.433195    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1d51474-e0df-472a-a8ba-145e29d62d3c-metrics-certs" pod "f1d51474-e0df-472a-a8ba-145e29d62d3c" (UID: "f1d51474-e0df-472a-a8ba-145e29d62d3c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:24.433339 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:24.433210    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1d51474-e0df-472a-a8ba-145e29d62d3c-maas-api-tls" pod "f1d51474-e0df-472a-a8ba-145e29d62d3c" (UID: "f1d51474-e0df-472a-a8ba-145e29d62d3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:14:24.532629 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:24.532601    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d51474-e0df-472a-a8ba-145e29d62d3c-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:14:24.532629 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:24.532624    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvcsj\" (UniqueName: \"kubernetes.io/projected/f1d51474-e0df-472a-a8ba-145e29d62d3c-kube-api-access-nvcsj\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:14:24.532629 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:24.532633    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1d51474-e0df-472a-a8ba-145e29d62d3c-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:14:24.532817 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:24.532642    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1d51474-e0df-472a-a8ba-145e29d62d3c-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:14:25.191677 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:25.191603    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv"
Oct 30 04:14:25.191677 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:25.191618    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv" event={"ID":"f1d51474-e0df-472a-a8ba-145e29d62d3c","Type":"ContainerDied","Data":"c50d852248a2c532d628505c43865ee3f8a28434585b8d105de5c6f1b1a1049d"}
Oct 30 04:14:25.191677 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:25.191676    4874 scope.go:122] "RemoveContainer" containerID="aee4c92d1b4e4dda967e09b23fcc3567e03ab98e37d0dc09c932ed926612a4d3"
Oct 30 04:14:25.239498 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:25.239473    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv"]
Oct 30 04:14:25.250637 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:25.250605    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-546d67dc88-pjqfv"]
Oct 30 04:14:25.607706 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:14:25.607674    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:14:26.726879 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:26.726844    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1d51474-e0df-472a-a8ba-145e29d62d3c" path="/var/lib/kubelet/pods/f1d51474-e0df-472a-a8ba-145e29d62d3c/volumes"
Oct 30 04:14:33.479053 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.479019    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"]
Oct 30 04:14:33.479567 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.479359    4874 cpu_manager.go:414] "containerMap: removing container" podUID="f1d51474-e0df-472a-a8ba-145e29d62d3c" containerName="maas-api"
Oct 30 04:14:33.479567 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.479372    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d51474-e0df-472a-a8ba-145e29d62d3c" containerName="maas-api"
Oct 30 04:14:33.479660 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.479593    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d51474-e0df-472a-a8ba-145e29d62d3c" containerName="maas-api"
Oct 30 04:14:33.483942 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.483925    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.501769 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.501748    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-vwdzb\""
Oct 30 04:14:33.513616 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.513590    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"]
Oct 30 04:14:33.598061 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.598035    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.598183 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.598064    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.598183 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.598090    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.598278 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.598200    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.598314 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.598284    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.598349 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.598314    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpcfm\" (UniqueName: \"kubernetes.io/projected/3cff5f9b-18b8-45b7-8052-09df41a8d91d-kube-api-access-qpcfm\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.598394 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.598377    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.598430 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.598416    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.598497 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.598485    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.698983 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.698959    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.698983 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.698986    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.699260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.699015    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.699260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.699039    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.699260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.699068    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.699260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.699115    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpcfm\" (UniqueName: \"kubernetes.io/projected/3cff5f9b-18b8-45b7-8052-09df41a8d91d-kube-api-access-qpcfm\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.699260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.699150    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.699260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.699173    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.699260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.699221    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.699673 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.699552    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.699673 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.699630    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.699873 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.699850    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.699933 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.699869    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.699933 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.699862    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.701590 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.701569    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.701858 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.701835    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.727509 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.727484    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.732269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.732195    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpcfm\" (UniqueName: \"kubernetes.io/projected/3cff5f9b-18b8-45b7-8052-09df41a8d91d-kube-api-access-qpcfm\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sdbgt\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:33.796151 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:33.796125    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:34.106791 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:34.106759    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"]
Oct 30 04:14:34.108271 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:14:34.108227    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cff5f9b_18b8_45b7_8052_09df41a8d91d.slice/crio-a925cfc11520e46b1856d2e0fe3877d4183a7d493806a6a4a83953c4859d46e2 WatchSource:0}: Error finding container a925cfc11520e46b1856d2e0fe3877d4183a7d493806a6a4a83953c4859d46e2: Status 404 returned error can't find the container with id a925cfc11520e46b1856d2e0fe3877d4183a7d493806a6a4a83953c4859d46e2
Oct 30 04:14:34.231634 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:34.231601    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt" event={"ID":"3cff5f9b-18b8-45b7-8052-09df41a8d91d","Type":"ContainerStarted","Data":"a925cfc11520e46b1856d2e0fe3877d4183a7d493806a6a4a83953c4859d46e2"}
Oct 30 04:14:36.684532 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:36.684504    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:14:36.684788 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:36.684567    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:14:36.684788 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:36.684597    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:14:37.244180 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:37.244140    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt" event={"ID":"3cff5f9b-18b8-45b7-8052-09df41a8d91d","Type":"ContainerStarted","Data":"c7dc240dd32b78a1d2f6040935d7ccb202e068e54bb3ea13b0697a131e86514c"}
Oct 30 04:14:37.310479 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:37.310421    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt" podStartSLOduration=1.736717573 podStartE2EDuration="4.310407115s" totalImagesPullingTime="2.573689542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:37.309174603 +0000 UTC m=+996.721570199" watchObservedRunningTime="2026-10-30 04:14:37.310407115 +0000 UTC m=+996.722802687"
Oct 30 04:14:37.796664 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:37.796616    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:37.798081 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:37.798054    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-sdbgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Oct 30 04:14:37.798214 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:37.798103    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt" podUID="3cff5f9b-18b8-45b7-8052-09df41a8d91d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Oct 30 04:14:38.797514 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:38.797474    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-sdbgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Oct 30 04:14:38.797895 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:38.797529    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt" podUID="3cff5f9b-18b8-45b7-8052-09df41a8d91d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Oct 30 04:14:39.796758 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:39.796720    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-sdbgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Oct 30 04:14:39.796932 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:39.796778    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt" podUID="3cff5f9b-18b8-45b7-8052-09df41a8d91d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Oct 30 04:14:40.797215 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:40.797180    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-sdbgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Oct 30 04:14:40.797680 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:40.797230    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt" podUID="3cff5f9b-18b8-45b7-8052-09df41a8d91d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Oct 30 04:14:41.797136 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:41.797105    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-sdbgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Oct 30 04:14:41.797333 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:41.797154    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt" podUID="3cff5f9b-18b8-45b7-8052-09df41a8d91d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Oct 30 04:14:42.800471 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:42.800441    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:42.800843 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:42.800822    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:42.801503 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:42.801484    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:14:48.527044 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.527004    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"]
Oct 30 04:14:48.530157 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.530129    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.560632 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.560610    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"]
Oct 30 04:14:48.564449 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.564428    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"]
Oct 30 04:14:48.567725 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.567703    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.588985 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.588965    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"]
Oct 30 04:14:48.637898 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.637872    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.638030 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.637914    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.638030 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.637936    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.638146 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.638063    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.638209 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.638174    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.638268 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.638230    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.638307 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.638284    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l8wr\" (UniqueName: \"kubernetes.io/projected/e22690dd-4871-45e2-b49e-8aaf73f19ae8-kube-api-access-5l8wr\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.638307 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.638303    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.638380 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.638320    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.739538 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.739510    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l8wr\" (UniqueName: \"kubernetes.io/projected/e22690dd-4871-45e2-b49e-8aaf73f19ae8-kube-api-access-5l8wr\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.739647 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.739540    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.739647 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.739556    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.739647 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.739576    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.739647 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.739598    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.739647 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.739617    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.739839 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.739742    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.739839 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.739797    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.739839 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.739829    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.739964 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.739861    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.739964 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.739879    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.739964 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.739903    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf576\" (UniqueName: \"kubernetes.io/projected/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-kube-api-access-sf576\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.740117 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.740103    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.740171 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.740116    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.740171 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.740133    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.740334 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.740212    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.740334 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.740220    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.740334 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.740315    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.740450 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.740332    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.740450 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.740349    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.740450 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.740382    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.740450 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.740417    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.740580 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.740562    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.742069 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.742047    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.742577 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.742553    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.761936 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.761903    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.762051 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.762034    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l8wr\" (UniqueName: \"kubernetes.io/projected/e22690dd-4871-45e2-b49e-8aaf73f19ae8-kube-api-access-5l8wr\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xfldb\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.841607 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.841538    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.841607 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.841579    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.841607 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.841603    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.841853 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.841638    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.841853 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.841668    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.841853 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.841704    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.841853 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.841736    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf576\" (UniqueName: \"kubernetes.io/projected/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-kube-api-access-sf576\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.841853 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.841771    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.841853 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.841798    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.841853 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.841812    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:48.842189 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.842060    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.842265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.842187    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.842352 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.842317    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.842470 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.842406    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.842554 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.842534    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.843776 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.843752    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.844195 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.844179    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.863965 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.863944    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.864601 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.864574    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf576\" (UniqueName: \"kubernetes.io/projected/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-kube-api-access-sf576\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8gd4\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:48.881235 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:48.881214    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:49.152611 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:49.152589    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"]
Oct 30 04:14:49.154028 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:14:49.153999    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode22690dd_4871_45e2_b49e_8aaf73f19ae8.slice/crio-55eea014dff3b8d8e395faf8310ea39bf78b1059d70377b0f88a06fecc8114e2 WatchSource:0}: Error finding container 55eea014dff3b8d8e395faf8310ea39bf78b1059d70377b0f88a06fecc8114e2: Status 404 returned error can't find the container with id 55eea014dff3b8d8e395faf8310ea39bf78b1059d70377b0f88a06fecc8114e2
Oct 30 04:14:49.156896 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:49.156868    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:14:49.156945 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:49.156929    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:14:49.156987 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:49.156955    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:14:49.193032 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:49.193013    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"]
Oct 30 04:14:49.194553 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:14:49.194530    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d0f966_46ee_4f8a_9caa_97c985bcd2a8.slice/crio-aefc16e66959da51a8d9f9df0987d8accd8b1b0225cb25e2109118040257e062 WatchSource:0}: Error finding container aefc16e66959da51a8d9f9df0987d8accd8b1b0225cb25e2109118040257e062: Status 404 returned error can't find the container with id aefc16e66959da51a8d9f9df0987d8accd8b1b0225cb25e2109118040257e062
Oct 30 04:14:49.197815 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:49.197779    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:14:49.197894 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:49.197868    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:14:49.197939 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:49.197915    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:14:49.325121 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:49.324264    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4" event={"ID":"81d0f966-46ee-4f8a-9caa-97c985bcd2a8","Type":"ContainerStarted","Data":"37a6ca78766e2559ee222922676d8733b32b08173c5297c97e18668b41036787"}
Oct 30 04:14:49.325121 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:49.324337    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4" event={"ID":"81d0f966-46ee-4f8a-9caa-97c985bcd2a8","Type":"ContainerStarted","Data":"aefc16e66959da51a8d9f9df0987d8accd8b1b0225cb25e2109118040257e062"}
Oct 30 04:14:49.328202 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:49.327616    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb" event={"ID":"e22690dd-4871-45e2-b49e-8aaf73f19ae8","Type":"ContainerStarted","Data":"eb9d851a1fb28e63f92adb000881097ae7e61c5a376837b87e49725af40c65fe"}
Oct 30 04:14:49.328202 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:49.327670    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb" event={"ID":"e22690dd-4871-45e2-b49e-8aaf73f19ae8","Type":"ContainerStarted","Data":"55eea014dff3b8d8e395faf8310ea39bf78b1059d70377b0f88a06fecc8114e2"}
Oct 30 04:14:49.370059 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:49.369988    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4" podStartSLOduration=1.369968734 podStartE2EDuration="1.369968734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:49.368430606 +0000 UTC m=+1008.780826206" watchObservedRunningTime="2026-10-30 04:14:49.369968734 +0000 UTC m=+1008.782364307"
Oct 30 04:14:49.412032 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:49.411919    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb" podStartSLOduration=1.411905508 podStartE2EDuration="1.411905508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:49.411370003 +0000 UTC m=+1008.823765779" watchObservedRunningTime="2026-10-30 04:14:49.411905508 +0000 UTC m=+1008.824301080"
Oct 30 04:14:49.842567 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:49.842522    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:49.881425 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:49.881337    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:50.843854 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:50.843816    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-xfldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body=
Oct 30 04:14:50.844263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:50.843868    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb" podUID="e22690dd-4871-45e2-b49e-8aaf73f19ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused"
Oct 30 04:14:50.890271 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:50.889525    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-x8gd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body=
Oct 30 04:14:50.890271 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:50.889579    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4" podUID="81d0f966-46ee-4f8a-9caa-97c985bcd2a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused"
Oct 30 04:14:51.842754 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:51.842712    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-xfldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body=
Oct 30 04:14:51.843031 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:51.843001    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb" podUID="e22690dd-4871-45e2-b49e-8aaf73f19ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused"
Oct 30 04:14:51.883881 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:51.883847    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-x8gd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body=
Oct 30 04:14:51.884497 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:51.884469    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4" podUID="81d0f966-46ee-4f8a-9caa-97c985bcd2a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused"
Oct 30 04:14:52.843429 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:52.843390    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-xfldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body=
Oct 30 04:14:52.843636 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:52.843452    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb" podUID="e22690dd-4871-45e2-b49e-8aaf73f19ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused"
Oct 30 04:14:52.883348 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:52.883306    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-x8gd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body=
Oct 30 04:14:52.883868 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:52.883590    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4" podUID="81d0f966-46ee-4f8a-9caa-97c985bcd2a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused"
Oct 30 04:14:53.842783 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:53.842744    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-xfldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body=
Oct 30 04:14:53.843146 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:53.842797    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb" podUID="e22690dd-4871-45e2-b49e-8aaf73f19ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused"
Oct 30 04:14:53.884787 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:53.884741    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-x8gd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body=
Oct 30 04:14:53.885060 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:53.885022    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4" podUID="81d0f966-46ee-4f8a-9caa-97c985bcd2a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused"
Oct 30 04:14:54.842581 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:54.842543    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-xfldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body=
Oct 30 04:14:54.842759 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:54.842595    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb" podUID="e22690dd-4871-45e2-b49e-8aaf73f19ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused"
Oct 30 04:14:54.884263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:54.882969    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-x8gd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body=
Oct 30 04:14:54.884263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:54.883051    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4" podUID="81d0f966-46ee-4f8a-9caa-97c985bcd2a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused"
Oct 30 04:14:55.608023 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:14:55.607975    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:14:55.847118 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:55.847084    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:55.847494 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:55.847472    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:55.848064 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:55.848045    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:14:55.885307 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:55.885248    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:55.885676 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:55.885509    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:14:55.886161 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:14:55.886136    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:15:25.608312 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:15:25.608221    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:15:30.374185 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:30.374139    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:15:35.314991 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.314956    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:15:35.350334 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.350304    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"]
Oct 30 04:15:35.353871 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.353849    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"
Oct 30 04:15:35.378949 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.378927    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dx86c\""
Oct 30 04:15:35.379072 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.379019    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 04:15:35.379072 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.379019    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 04:15:35.379147 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.379092    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 04:15:35.380562 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.380534    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"]
Oct 30 04:15:35.499268 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.499225    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d775e3cc-9ee3-4b5c-a155-e26d1a854eab-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l8nmh\" (UID: \"d775e3cc-9ee3-4b5c-a155-e26d1a854eab\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"
Oct 30 04:15:35.499404 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.499330    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d775e3cc-9ee3-4b5c-a155-e26d1a854eab-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l8nmh\" (UID: \"d775e3cc-9ee3-4b5c-a155-e26d1a854eab\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"
Oct 30 04:15:35.499448 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.499404    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d775e3cc-9ee3-4b5c-a155-e26d1a854eab-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l8nmh\" (UID: \"d775e3cc-9ee3-4b5c-a155-e26d1a854eab\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"
Oct 30 04:15:35.499448 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.499438    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d775e3cc-9ee3-4b5c-a155-e26d1a854eab-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l8nmh\" (UID: \"d775e3cc-9ee3-4b5c-a155-e26d1a854eab\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"
Oct 30 04:15:35.499523 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.499510    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d775e3cc-9ee3-4b5c-a155-e26d1a854eab-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l8nmh\" (UID: \"d775e3cc-9ee3-4b5c-a155-e26d1a854eab\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"
Oct 30 04:15:35.499557 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.499542    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vblw\" (UniqueName: \"kubernetes.io/projected/d775e3cc-9ee3-4b5c-a155-e26d1a854eab-kube-api-access-8vblw\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l8nmh\" (UID: \"d775e3cc-9ee3-4b5c-a155-e26d1a854eab\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"
Oct 30 04:15:35.600220 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.600139    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d775e3cc-9ee3-4b5c-a155-e26d1a854eab-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l8nmh\" (UID: \"d775e3cc-9ee3-4b5c-a155-e26d1a854eab\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"
Oct 30 04:15:35.600220 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.600195    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d775e3cc-9ee3-4b5c-a155-e26d1a854eab-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l8nmh\" (UID: \"d775e3cc-9ee3-4b5c-a155-e26d1a854eab\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"
Oct 30 04:15:35.600220 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.600220    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d775e3cc-9ee3-4b5c-a155-e26d1a854eab-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l8nmh\" (UID: \"d775e3cc-9ee3-4b5c-a155-e26d1a854eab\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"
Oct 30 04:15:35.600495 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.600250    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d775e3cc-9ee3-4b5c-a155-e26d1a854eab-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l8nmh\" (UID: \"d775e3cc-9ee3-4b5c-a155-e26d1a854eab\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"
Oct 30 04:15:35.600495 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.600274    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d775e3cc-9ee3-4b5c-a155-e26d1a854eab-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l8nmh\" (UID: \"d775e3cc-9ee3-4b5c-a155-e26d1a854eab\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"
Oct 30 04:15:35.600495 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.600307    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vblw\" (UniqueName: \"kubernetes.io/projected/d775e3cc-9ee3-4b5c-a155-e26d1a854eab-kube-api-access-8vblw\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l8nmh\" (UID: \"d775e3cc-9ee3-4b5c-a155-e26d1a854eab\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"
Oct 30 04:15:35.600657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.600611    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d775e3cc-9ee3-4b5c-a155-e26d1a854eab-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l8nmh\" (UID: \"d775e3cc-9ee3-4b5c-a155-e26d1a854eab\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"
Oct 30 04:15:35.600657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.600644    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d775e3cc-9ee3-4b5c-a155-e26d1a854eab-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l8nmh\" (UID: \"d775e3cc-9ee3-4b5c-a155-e26d1a854eab\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"
Oct 30 04:15:35.600767 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.600741    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d775e3cc-9ee3-4b5c-a155-e26d1a854eab-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l8nmh\" (UID: \"d775e3cc-9ee3-4b5c-a155-e26d1a854eab\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"
Oct 30 04:15:35.602456 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.602434    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d775e3cc-9ee3-4b5c-a155-e26d1a854eab-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l8nmh\" (UID: \"d775e3cc-9ee3-4b5c-a155-e26d1a854eab\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"
Oct 30 04:15:35.602724 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.602708    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d775e3cc-9ee3-4b5c-a155-e26d1a854eab-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l8nmh\" (UID: \"d775e3cc-9ee3-4b5c-a155-e26d1a854eab\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"
Oct 30 04:15:35.622349 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.622327    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vblw\" (UniqueName: \"kubernetes.io/projected/d775e3cc-9ee3-4b5c-a155-e26d1a854eab-kube-api-access-8vblw\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l8nmh\" (UID: \"d775e3cc-9ee3-4b5c-a155-e26d1a854eab\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"
Oct 30 04:15:35.666516 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.666491    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"
Oct 30 04:15:35.962536 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:35.962502    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"]
Oct 30 04:15:35.963749 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:15:35.963721    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd775e3cc_9ee3_4b5c_a155_e26d1a854eab.slice/crio-1d4f4401a4a77e23ed5fdb41eae5cb1f9fb2b505bbfd42ecd5f79c40a0ce0f68 WatchSource:0}: Error finding container 1d4f4401a4a77e23ed5fdb41eae5cb1f9fb2b505bbfd42ecd5f79c40a0ce0f68: Status 404 returned error can't find the container with id 1d4f4401a4a77e23ed5fdb41eae5cb1f9fb2b505bbfd42ecd5f79c40a0ce0f68
Oct 30 04:15:36.525092 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:36.525058    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh" event={"ID":"d775e3cc-9ee3-4b5c-a155-e26d1a854eab","Type":"ContainerStarted","Data":"1d4f4401a4a77e23ed5fdb41eae5cb1f9fb2b505bbfd42ecd5f79c40a0ce0f68"}
Oct 30 04:15:37.538604 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:37.538564    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh" event={"ID":"d775e3cc-9ee3-4b5c-a155-e26d1a854eab","Type":"ContainerStarted","Data":"434a4337a2a0a30e6b88dc457ff187fe9657aac09a0de308a38a989bdafc3493"}
Oct 30 04:15:37.586261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:37.585268    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh" podStartSLOduration=1.446638481 podStartE2EDuration="2.585233667s" totalImagesPullingTime="1.138595186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:37.583703238 +0000 UTC m=+1056.996098834" watchObservedRunningTime="2026-10-30 04:15:37.585233667 +0000 UTC m=+1056.997629241"
Oct 30 04:15:37.669109 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:37.669061    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"
Oct 30 04:15:39.246622 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:39.246589    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:15:39.708932 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:39.708840    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"]
Oct 30 04:15:39.712756 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:39.712731    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"
Oct 30 04:15:39.735567 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:39.735537    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 04:15:39.748674 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:39.748629    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"]
Oct 30 04:15:39.777293 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:39.777262    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522d3048-9568-4813-b2ad-69f2ea7defb8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp\" (UID: \"522d3048-9568-4813-b2ad-69f2ea7defb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"
Oct 30 04:15:39.777293 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:39.777302    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522d3048-9568-4813-b2ad-69f2ea7defb8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp\" (UID: \"522d3048-9568-4813-b2ad-69f2ea7defb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"
Oct 30 04:15:39.777543 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:39.777326    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/522d3048-9568-4813-b2ad-69f2ea7defb8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp\" (UID: \"522d3048-9568-4813-b2ad-69f2ea7defb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"
Oct 30 04:15:39.777543 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:39.777433    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lltgw\" (UniqueName: \"kubernetes.io/projected/522d3048-9568-4813-b2ad-69f2ea7defb8-kube-api-access-lltgw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp\" (UID: \"522d3048-9568-4813-b2ad-69f2ea7defb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"
Oct 30 04:15:39.777543 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:39.777478    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/522d3048-9568-4813-b2ad-69f2ea7defb8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp\" (UID: \"522d3048-9568-4813-b2ad-69f2ea7defb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"
Oct 30 04:15:39.777730 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:39.777559    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/522d3048-9568-4813-b2ad-69f2ea7defb8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp\" (UID: \"522d3048-9568-4813-b2ad-69f2ea7defb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"
Oct 30 04:15:39.878820 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:39.878773    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522d3048-9568-4813-b2ad-69f2ea7defb8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp\" (UID: \"522d3048-9568-4813-b2ad-69f2ea7defb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"
Oct 30 04:15:39.879013 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:39.878847    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522d3048-9568-4813-b2ad-69f2ea7defb8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp\" (UID: \"522d3048-9568-4813-b2ad-69f2ea7defb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"
Oct 30 04:15:39.879013 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:39.878881    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/522d3048-9568-4813-b2ad-69f2ea7defb8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp\" (UID: \"522d3048-9568-4813-b2ad-69f2ea7defb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"
Oct 30 04:15:39.879013 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:39.878918    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lltgw\" (UniqueName: \"kubernetes.io/projected/522d3048-9568-4813-b2ad-69f2ea7defb8-kube-api-access-lltgw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp\" (UID: \"522d3048-9568-4813-b2ad-69f2ea7defb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"
Oct 30 04:15:39.879013 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:39.878963    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/522d3048-9568-4813-b2ad-69f2ea7defb8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp\" (UID: \"522d3048-9568-4813-b2ad-69f2ea7defb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"
Oct 30 04:15:39.879269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:39.879049    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/522d3048-9568-4813-b2ad-69f2ea7defb8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp\" (UID: \"522d3048-9568-4813-b2ad-69f2ea7defb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"
Oct 30 04:15:39.879361 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:39.879335    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522d3048-9568-4813-b2ad-69f2ea7defb8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp\" (UID: \"522d3048-9568-4813-b2ad-69f2ea7defb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"
Oct 30 04:15:39.879762 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:39.879738    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522d3048-9568-4813-b2ad-69f2ea7defb8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp\" (UID: \"522d3048-9568-4813-b2ad-69f2ea7defb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"
Oct 30 04:15:39.879961 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:39.879770    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/522d3048-9568-4813-b2ad-69f2ea7defb8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp\" (UID: \"522d3048-9568-4813-b2ad-69f2ea7defb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"
Oct 30 04:15:39.882069 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:39.882042    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/522d3048-9568-4813-b2ad-69f2ea7defb8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp\" (UID: \"522d3048-9568-4813-b2ad-69f2ea7defb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"
Oct 30 04:15:39.882325 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:39.882301    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/522d3048-9568-4813-b2ad-69f2ea7defb8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp\" (UID: \"522d3048-9568-4813-b2ad-69f2ea7defb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"
Oct 30 04:15:39.904741 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:39.904706    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lltgw\" (UniqueName: \"kubernetes.io/projected/522d3048-9568-4813-b2ad-69f2ea7defb8-kube-api-access-lltgw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp\" (UID: \"522d3048-9568-4813-b2ad-69f2ea7defb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"
Oct 30 04:15:40.046787 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:40.046420    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"
Oct 30 04:15:40.645589 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:40.644936    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"]
Oct 30 04:15:40.646923 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:15:40.646885    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522d3048_9568_4813_b2ad_69f2ea7defb8.slice/crio-93cd3163640792af98d91fa51a417b6f0f458e2ff586c138b7655bed01dd56c4 WatchSource:0}: Error finding container 93cd3163640792af98d91fa51a417b6f0f458e2ff586c138b7655bed01dd56c4: Status 404 returned error can't find the container with id 93cd3163640792af98d91fa51a417b6f0f458e2ff586c138b7655bed01dd56c4
Oct 30 04:15:41.558555 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:41.558514    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp" event={"ID":"522d3048-9568-4813-b2ad-69f2ea7defb8","Type":"ContainerStarted","Data":"6579f559a4ff4fcc5778dcea853f9adf7b8c257423852971217e0cd22f0d69f6"}
Oct 30 04:15:41.558752 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:41.558592    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp" event={"ID":"522d3048-9568-4813-b2ad-69f2ea7defb8","Type":"ContainerStarted","Data":"93cd3163640792af98d91fa51a417b6f0f458e2ff586c138b7655bed01dd56c4"}
Oct 30 04:15:41.599788 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:41.599740    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp" podStartSLOduration=2.365513448 podStartE2EDuration="2.599727901s" totalImagesPullingTime="234.214453ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:41.598847381 +0000 UTC m=+1061.011242953" watchObservedRunningTime="2026-10-30 04:15:41.599727901 +0000 UTC m=+1061.012123474"
Oct 30 04:15:42.047175 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:42.047137    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"
Oct 30 04:15:42.294217 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:42.294181    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-n6b8k"]
Oct 30 04:15:42.297852 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:42.297781    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k"
Oct 30 04:15:42.323646 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:42.323620    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 30 04:15:42.325356 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:42.325329    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-n6b8k"]
Oct 30 04:15:42.431209 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:42.431170    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmfq2\" (UniqueName: \"kubernetes.io/projected/c2f77027-4d74-47c0-8593-560a26c071e9-kube-api-access-tmfq2\") pod \"echo-simulator-kserve-64b6f8f96c-n6b8k\" (UID: \"c2f77027-4d74-47c0-8593-560a26c071e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k"
Oct 30 04:15:42.431209 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:42.431223    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2f77027-4d74-47c0-8593-560a26c071e9-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-n6b8k\" (UID: \"c2f77027-4d74-47c0-8593-560a26c071e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k"
Oct 30 04:15:42.431457 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:42.431336    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2f77027-4d74-47c0-8593-560a26c071e9-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-n6b8k\" (UID: \"c2f77027-4d74-47c0-8593-560a26c071e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k"
Oct 30 04:15:42.431457 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:42.431370    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2f77027-4d74-47c0-8593-560a26c071e9-home\") pod \"echo-simulator-kserve-64b6f8f96c-n6b8k\" (UID: \"c2f77027-4d74-47c0-8593-560a26c071e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k"
Oct 30 04:15:42.431457 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:42.431441    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2f77027-4d74-47c0-8593-560a26c071e9-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-n6b8k\" (UID: \"c2f77027-4d74-47c0-8593-560a26c071e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k"
Oct 30 04:15:42.431605 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:42.431502    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2f77027-4d74-47c0-8593-560a26c071e9-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-n6b8k\" (UID: \"c2f77027-4d74-47c0-8593-560a26c071e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k"
Oct 30 04:15:42.532071 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:42.532029    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2f77027-4d74-47c0-8593-560a26c071e9-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-n6b8k\" (UID: \"c2f77027-4d74-47c0-8593-560a26c071e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k"
Oct 30 04:15:42.532273 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:42.532098    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmfq2\" (UniqueName: \"kubernetes.io/projected/c2f77027-4d74-47c0-8593-560a26c071e9-kube-api-access-tmfq2\") pod \"echo-simulator-kserve-64b6f8f96c-n6b8k\" (UID: \"c2f77027-4d74-47c0-8593-560a26c071e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k"
Oct 30 04:15:42.532273 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:42.532139    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2f77027-4d74-47c0-8593-560a26c071e9-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-n6b8k\" (UID: \"c2f77027-4d74-47c0-8593-560a26c071e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k"
Oct 30 04:15:42.532273 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:42.532197    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2f77027-4d74-47c0-8593-560a26c071e9-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-n6b8k\" (UID: \"c2f77027-4d74-47c0-8593-560a26c071e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k"
Oct 30 04:15:42.532273 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:42.532256    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2f77027-4d74-47c0-8593-560a26c071e9-home\") pod \"echo-simulator-kserve-64b6f8f96c-n6b8k\" (UID: \"c2f77027-4d74-47c0-8593-560a26c071e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k"
Oct 30 04:15:42.532515 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:42.532282    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2f77027-4d74-47c0-8593-560a26c071e9-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-n6b8k\" (UID: \"c2f77027-4d74-47c0-8593-560a26c071e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k"
Oct 30 04:15:42.533127 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:42.533098    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2f77027-4d74-47c0-8593-560a26c071e9-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-n6b8k\" (UID: \"c2f77027-4d74-47c0-8593-560a26c071e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k"
Oct 30 04:15:42.533374 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:42.533350    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2f77027-4d74-47c0-8593-560a26c071e9-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-n6b8k\" (UID: \"c2f77027-4d74-47c0-8593-560a26c071e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k"
Oct 30 04:15:42.533874 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:42.533843    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2f77027-4d74-47c0-8593-560a26c071e9-home\") pod \"echo-simulator-kserve-64b6f8f96c-n6b8k\" (UID: \"c2f77027-4d74-47c0-8593-560a26c071e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k"
Oct 30 04:15:42.538347 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:42.538293    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2f77027-4d74-47c0-8593-560a26c071e9-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-n6b8k\" (UID: \"c2f77027-4d74-47c0-8593-560a26c071e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k"
Oct 30 04:15:42.538454 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:42.538404    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2f77027-4d74-47c0-8593-560a26c071e9-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-n6b8k\" (UID: \"c2f77027-4d74-47c0-8593-560a26c071e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k"
Oct 30 04:15:42.556606 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:42.556534    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmfq2\" (UniqueName: \"kubernetes.io/projected/c2f77027-4d74-47c0-8593-560a26c071e9-kube-api-access-tmfq2\") pod \"echo-simulator-kserve-64b6f8f96c-n6b8k\" (UID: \"c2f77027-4d74-47c0-8593-560a26c071e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k"
Oct 30 04:15:42.611074 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:42.611027    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k"
Oct 30 04:15:42.946319 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:42.941826    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-n6b8k"]
Oct 30 04:15:43.627382 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:43.627308    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k" event={"ID":"c2f77027-4d74-47c0-8593-560a26c071e9","Type":"ContainerStarted","Data":"1c7d2349a99c2cfb62194d237700a89194fefd1d3d23894f41fa31f5d0cec077"}
Oct 30 04:15:44.727975 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:44.727940    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:15:45.673263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:45.670728    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"
Oct 30 04:15:45.679148 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:45.678487    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k" event={"ID":"c2f77027-4d74-47c0-8593-560a26c071e9","Type":"ContainerStarted","Data":"8928adf828d1dd9e81ec34fa8b94520ce735e9f396548ba42c3929dccf11d4af"}
Oct 30 04:15:45.700689 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:45.700621    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"
Oct 30 04:15:45.725527 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:45.724814    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k" podStartSLOduration=1.490731394 podStartE2EDuration="3.724796484s" totalImagesPullingTime="2.23406509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:45.724734765 +0000 UTC m=+1065.137130340" watchObservedRunningTime="2026-10-30 04:15:45.724796484 +0000 UTC m=+1065.137192059"
Oct 30 04:15:46.616279 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:46.613324    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k"
Oct 30 04:15:46.706844 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:46.706818    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l8nmh"
Oct 30 04:15:50.047079 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:50.047044    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"
Oct 30 04:15:50.059556 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:50.059528    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"
Oct 30 04:15:50.753442 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:50.753414    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp"
Oct 30 04:15:51.246123 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:51.246084    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:15:52.612300 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:52.612260    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k"
Oct 30 04:15:52.631986 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:52.631958    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k"
Oct 30 04:15:52.769533 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:15:52.769500    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-n6b8k"
Oct 30 04:15:55.607743 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:15:55.607704    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:16:07.131095 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:07.131061    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-757f4bf64c-szwjz"]
Oct 30 04:16:07.134478 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:07.134459    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-757f4bf64c-szwjz"
Oct 30 04:16:07.152315 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:07.152292    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-757f4bf64c-szwjz"]
Oct 30 04:16:07.238697 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:07.238664    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55d7m\" (UniqueName: \"kubernetes.io/projected/36ea66f0-e11f-4da2-a421-1498c0287776-kube-api-access-55d7m\") pod \"authorino-757f4bf64c-szwjz\" (UID: \"36ea66f0-e11f-4da2-a421-1498c0287776\") " pod="kuadrant-system/authorino-757f4bf64c-szwjz"
Oct 30 04:16:07.238849 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:07.238791    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36ea66f0-e11f-4da2-a421-1498c0287776-tls-cert\") pod \"authorino-757f4bf64c-szwjz\" (UID: \"36ea66f0-e11f-4da2-a421-1498c0287776\") " pod="kuadrant-system/authorino-757f4bf64c-szwjz"
Oct 30 04:16:07.339408 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:07.339375    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36ea66f0-e11f-4da2-a421-1498c0287776-tls-cert\") pod \"authorino-757f4bf64c-szwjz\" (UID: \"36ea66f0-e11f-4da2-a421-1498c0287776\") " pod="kuadrant-system/authorino-757f4bf64c-szwjz"
Oct 30 04:16:07.339556 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:07.339419    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55d7m\" (UniqueName: \"kubernetes.io/projected/36ea66f0-e11f-4da2-a421-1498c0287776-kube-api-access-55d7m\") pod \"authorino-757f4bf64c-szwjz\" (UID: \"36ea66f0-e11f-4da2-a421-1498c0287776\") " pod="kuadrant-system/authorino-757f4bf64c-szwjz"
Oct 30 04:16:07.341866 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:07.341841    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36ea66f0-e11f-4da2-a421-1498c0287776-tls-cert\") pod \"authorino-757f4bf64c-szwjz\" (UID: \"36ea66f0-e11f-4da2-a421-1498c0287776\") " pod="kuadrant-system/authorino-757f4bf64c-szwjz"
Oct 30 04:16:07.361842 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:07.361814    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55d7m\" (UniqueName: \"kubernetes.io/projected/36ea66f0-e11f-4da2-a421-1498c0287776-kube-api-access-55d7m\") pod \"authorino-757f4bf64c-szwjz\" (UID: \"36ea66f0-e11f-4da2-a421-1498c0287776\") " pod="kuadrant-system/authorino-757f4bf64c-szwjz"
Oct 30 04:16:07.445953 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:07.445883    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-757f4bf64c-szwjz"
Oct 30 04:16:07.743578 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:07.743549    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-757f4bf64c-szwjz"]
Oct 30 04:16:07.744681 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:16:07.744656    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36ea66f0_e11f_4da2_a421_1498c0287776.slice/crio-e3e5759e14535149cabf9fd87d2f3508a8a5516a71957685dd481744dee86bcb WatchSource:0}: Error finding container e3e5759e14535149cabf9fd87d2f3508a8a5516a71957685dd481744dee86bcb: Status 404 returned error can't find the container with id e3e5759e14535149cabf9fd87d2f3508a8a5516a71957685dd481744dee86bcb
Oct 30 04:16:07.813434 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:07.813404    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-757f4bf64c-szwjz" event={"ID":"36ea66f0-e11f-4da2-a421-1498c0287776","Type":"ContainerStarted","Data":"e3e5759e14535149cabf9fd87d2f3508a8a5516a71957685dd481744dee86bcb"}
Oct 30 04:16:08.822113 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:08.822069    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-757f4bf64c-szwjz" event={"ID":"36ea66f0-e11f-4da2-a421-1498c0287776","Type":"ContainerStarted","Data":"ebfd31599cdec76984e453b5dfc4ac3d4c710e372619b2a823406465895e4ab6"}
Oct 30 04:16:08.875264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:08.873106    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-757f4bf64c-szwjz" podStartSLOduration=1.384160811 podStartE2EDuration="1.873086s" totalImagesPullingTime="488.925189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:08.872533803 +0000 UTC m=+1088.284929377" watchObservedRunningTime="2026-10-30 04:16:08.873086 +0000 UTC m=+1088.285481573"
Oct 30 04:16:08.888879 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:08.888837    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65f8dc9b64-t2xjq"]
Oct 30 04:16:08.889078 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:08.889052    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-65f8dc9b64-t2xjq" podUID="c44245be-88c9-4175-ae0b-257f5f4f425b" containerName="authorino" containerID="cri-o://6b72236a126623b76ec524bf9e2d4a12f6ff5339f2cc93ef61ed9ad28e6d0a7e" gracePeriod=30
Oct 30 04:16:09.158502 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:09.158472    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65f8dc9b64-t2xjq"
Oct 30 04:16:09.255603 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:09.255565    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c44245be-88c9-4175-ae0b-257f5f4f425b-tls-cert\" (UniqueName: \"kubernetes.io/secret/c44245be-88c9-4175-ae0b-257f5f4f425b-tls-cert\") pod \"c44245be-88c9-4175-ae0b-257f5f4f425b\" (UID: \"c44245be-88c9-4175-ae0b-257f5f4f425b\") "
Oct 30 04:16:09.255765 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:09.255617    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c44245be-88c9-4175-ae0b-257f5f4f425b-kube-api-access-g8vxl\" (UniqueName: \"kubernetes.io/projected/c44245be-88c9-4175-ae0b-257f5f4f425b-kube-api-access-g8vxl\") pod \"c44245be-88c9-4175-ae0b-257f5f4f425b\" (UID: \"c44245be-88c9-4175-ae0b-257f5f4f425b\") "
Oct 30 04:16:09.257675 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:09.257642    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44245be-88c9-4175-ae0b-257f5f4f425b-kube-api-access-g8vxl" pod "c44245be-88c9-4175-ae0b-257f5f4f425b" (UID: "c44245be-88c9-4175-ae0b-257f5f4f425b"). InnerVolumeSpecName "kube-api-access-g8vxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:09.267831 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:09.267803    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c44245be-88c9-4175-ae0b-257f5f4f425b-tls-cert" pod "c44245be-88c9-4175-ae0b-257f5f4f425b" (UID: "c44245be-88c9-4175-ae0b-257f5f4f425b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:09.357807 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:09.357725    4874 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c44245be-88c9-4175-ae0b-257f5f4f425b-tls-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:16:09.357807 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:09.357766    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8vxl\" (UniqueName: \"kubernetes.io/projected/c44245be-88c9-4175-ae0b-257f5f4f425b-kube-api-access-g8vxl\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:16:09.837259 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:09.836467    4874 generic.go:356] "Generic (PLEG): container finished" podID="c44245be-88c9-4175-ae0b-257f5f4f425b" containerID="6b72236a126623b76ec524bf9e2d4a12f6ff5339f2cc93ef61ed9ad28e6d0a7e" exitCode=0
Oct 30 04:16:09.837259 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:09.837123    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65f8dc9b64-t2xjq"
Oct 30 04:16:09.837801 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:09.837374    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65f8dc9b64-t2xjq" event={"ID":"c44245be-88c9-4175-ae0b-257f5f4f425b","Type":"ContainerDied","Data":"6b72236a126623b76ec524bf9e2d4a12f6ff5339f2cc93ef61ed9ad28e6d0a7e"}
Oct 30 04:16:09.837801 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:09.837420    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65f8dc9b64-t2xjq" event={"ID":"c44245be-88c9-4175-ae0b-257f5f4f425b","Type":"ContainerDied","Data":"1a3183871815f1010e09e4afbf441e2080a0886a8e2949363b3e1c211beccf2b"}
Oct 30 04:16:09.837801 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:09.837442    4874 scope.go:122] "RemoveContainer" containerID="6b72236a126623b76ec524bf9e2d4a12f6ff5339f2cc93ef61ed9ad28e6d0a7e"
Oct 30 04:16:09.848148 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:09.848128    4874 scope.go:122] "RemoveContainer" containerID="6b72236a126623b76ec524bf9e2d4a12f6ff5339f2cc93ef61ed9ad28e6d0a7e"
Oct 30 04:16:09.848459 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:16:09.848439    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b72236a126623b76ec524bf9e2d4a12f6ff5339f2cc93ef61ed9ad28e6d0a7e\": container with ID starting with 6b72236a126623b76ec524bf9e2d4a12f6ff5339f2cc93ef61ed9ad28e6d0a7e not found: ID does not exist" containerID="6b72236a126623b76ec524bf9e2d4a12f6ff5339f2cc93ef61ed9ad28e6d0a7e"
Oct 30 04:16:09.848542 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:09.848467    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b72236a126623b76ec524bf9e2d4a12f6ff5339f2cc93ef61ed9ad28e6d0a7e"} err="failed to get container status \"6b72236a126623b76ec524bf9e2d4a12f6ff5339f2cc93ef61ed9ad28e6d0a7e\": rpc error: code = NotFound desc = could not find container \"6b72236a126623b76ec524bf9e2d4a12f6ff5339f2cc93ef61ed9ad28e6d0a7e\": container with ID starting with 6b72236a126623b76ec524bf9e2d4a12f6ff5339f2cc93ef61ed9ad28e6d0a7e not found: ID does not exist"
Oct 30 04:16:09.887775 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:09.887748    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65f8dc9b64-t2xjq"]
Oct 30 04:16:09.894192 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:09.894163    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-65f8dc9b64-t2xjq"]
Oct 30 04:16:10.728343 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:10.728306    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c44245be-88c9-4175-ae0b-257f5f4f425b" path="/var/lib/kubelet/pods/c44245be-88c9-4175-ae0b-257f5f4f425b/volumes"
Oct 30 04:16:25.607935 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:16:25.607899    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:16:42.097746 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.097710    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"]
Oct 30 04:16:42.099099 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.099070    4874 cpu_manager.go:414] "containerMap: removing container" podUID="c44245be-88c9-4175-ae0b-257f5f4f425b" containerName="authorino"
Oct 30 04:16:42.099261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.099229    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c44245be-88c9-4175-ae0b-257f5f4f425b" containerName="authorino"
Oct 30 04:16:42.099522 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.099508    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="c44245be-88c9-4175-ae0b-257f5f4f425b" containerName="authorino"
Oct 30 04:16:42.103493 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.103461    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.117911 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.117883    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-de5733f9-openshift-default-dockercfg-d4t4g\""
Oct 30 04:16:42.130679 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.130655    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"]
Oct 30 04:16:42.145761 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.145737    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-data\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.145867 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.145777    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-workload-socket\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.145867 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.145804    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-podinfo\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.145970 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.145927    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/425e4e27-91f5-47b0-b8cf-311b5550e48a-istiod-ca-cert\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.145970 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.145947    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7l2g\" (UniqueName: \"kubernetes.io/projected/425e4e27-91f5-47b0-b8cf-311b5550e48a-kube-api-access-g7l2g\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.146081 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.145979    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-token\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.146081 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.145998    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-envoy\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.146195 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.146123    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-credential-socket\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.146195 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.146164    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-workload-certs\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.246739 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.246697    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"]
Oct 30 04:16:42.247068 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.247039    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-data\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.247181 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.247079    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-workload-socket\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.247181 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.247107    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-podinfo\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.247181 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.247170    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/425e4e27-91f5-47b0-b8cf-311b5550e48a-istiod-ca-cert\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.247375 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.247199    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7l2g\" (UniqueName: \"kubernetes.io/projected/425e4e27-91f5-47b0-b8cf-311b5550e48a-kube-api-access-g7l2g\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.247375 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.247234    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-token\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.247375 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.247279    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-envoy\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.247375 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.247304    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-credential-socket\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.247375 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.247333    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-workload-certs\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.247635 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.247525    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-data\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.247635 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.247561    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-workload-socket\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.247635 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.247609    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-workload-certs\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.248197 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.248166    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-credential-socket\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.248540 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.248516    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/425e4e27-91f5-47b0-b8cf-311b5550e48a-istiod-ca-cert\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.250068 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.250040    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-envoy\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.250221 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.250201    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-podinfo\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.263745 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.263727    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.270226 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.270206    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7l2g\" (UniqueName: \"kubernetes.io/projected/425e4e27-91f5-47b0-b8cf-311b5550e48a-kube-api-access-g7l2g\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.274603 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.274571    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-token\") pod \"e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") " pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.275210 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.275186    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"]
Oct 30 04:16:42.280009 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.279966    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-eb861bd9-openshift-default-dockercfg-7x9qb\""
Oct 30 04:16:42.348014 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.347946    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-data\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.348014 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.347982    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istiod-ca-cert\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.348014 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.348005    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-token\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.348218 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.348152    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-workload-socket\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.348218 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.348193    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-credential-socket\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.348218 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.348209    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-podinfo\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.348364 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.348235    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8x46\" (UniqueName: \"kubernetes.io/projected/35fb8fc8-50a2-409b-9c88-8ad5219ace64-kube-api-access-f8x46\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.348364 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.348332    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-envoy\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.348451 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.348371    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-workload-certs\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.416068 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.416042    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:42.449400 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.449371    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-workload-socket\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.449542 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.449419    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-credential-socket\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.449542 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.449436    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-podinfo\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.449542 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.449464    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8x46\" (UniqueName: \"kubernetes.io/projected/35fb8fc8-50a2-409b-9c88-8ad5219ace64-kube-api-access-f8x46\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.449542 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.449517    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-envoy\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.449754 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.449573    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-workload-certs\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.449754 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.449626    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-data\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.449754 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.449658    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istiod-ca-cert\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.449754 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.449685    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-token\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.450006 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.449810    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-credential-socket\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.450006 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.449842    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-workload-socket\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.450006 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.449874    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-workload-certs\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.450170 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.450068    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-data\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.450498 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.450477    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istiod-ca-cert\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.451781 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.451762    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-envoy\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.451993 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.451976    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-podinfo\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.497085 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.497015    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8x46\" (UniqueName: \"kubernetes.io/projected/35fb8fc8-50a2-409b-9c88-8ad5219ace64-kube-api-access-f8x46\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.499963 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.499739    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-token\") pod \"e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") " pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.592450 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.592418    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:42.634090 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.634048    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"]
Oct 30 04:16:42.639148 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.639122    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.651763 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.651734    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-istio-envoy\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.651878 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.651775    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-credential-socket\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.651878 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.651805    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-workload-certs\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.651878 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.651825    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-workload-socket\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.651878 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.651849    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-istio-data\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.652099 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.651882    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a09d87c-a553-4a74-8559-59141d1778a4-istio-token\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.652099 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.651900    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdm8r\" (UniqueName: \"kubernetes.io/projected/1a09d87c-a553-4a74-8559-59141d1778a4-kube-api-access-bdm8r\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.652099 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.651924    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a09d87c-a553-4a74-8559-59141d1778a4-istio-podinfo\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.652099 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.651938    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a09d87c-a553-4a74-8559-59141d1778a4-istiod-ca-cert\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.656155 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.656108    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-566c3724-gw-openshift-default-dockercfg-vtc7s\""
Oct 30 04:16:42.661500 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.661478    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"]
Oct 30 04:16:42.757774 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.754364    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-workload-socket\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.757774 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.754427    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-istio-data\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.757774 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.754486    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a09d87c-a553-4a74-8559-59141d1778a4-istio-token\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.757774 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.754513    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdm8r\" (UniqueName: \"kubernetes.io/projected/1a09d87c-a553-4a74-8559-59141d1778a4-kube-api-access-bdm8r\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.757774 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.754547    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a09d87c-a553-4a74-8559-59141d1778a4-istio-podinfo\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.757774 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.754574    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a09d87c-a553-4a74-8559-59141d1778a4-istiod-ca-cert\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.757774 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.754607    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-istio-envoy\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.757774 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.754643    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-credential-socket\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.757774 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.754699    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-workload-certs\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.757774 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.754884    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-workload-socket\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.757774 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.755035    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-workload-certs\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.757774 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.755303    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-credential-socket\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.769194 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.762607    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"]
Oct 30 04:16:42.769194 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.764751    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-istio-data\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.769194 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.764804    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a09d87c-a553-4a74-8559-59141d1778a4-istiod-ca-cert\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.769194 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.765619    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-istio-envoy\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.769194 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.769043    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a09d87c-a553-4a74-8559-59141d1778a4-istio-podinfo\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.772990 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:16:42.772963    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod425e4e27_91f5_47b0_b8cf_311b5550e48a.slice/crio-471821d323941e5cfe92e229fc3810316776c78f70a155ac71f95accc65b1a22 WatchSource:0}: Error finding container 471821d323941e5cfe92e229fc3810316776c78f70a155ac71f95accc65b1a22: Status 404 returned error can't find the container with id 471821d323941e5cfe92e229fc3810316776c78f70a155ac71f95accc65b1a22
Oct 30 04:16:42.775142 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.775112    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:16:42.775209 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.775178    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:16:42.775271 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.775207    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:16:42.786340 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.786279    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a09d87c-a553-4a74-8559-59141d1778a4-istio-token\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.786984 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.786967    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdm8r\" (UniqueName: \"kubernetes.io/projected/1a09d87c-a553-4a74-8559-59141d1778a4-kube-api-access-bdm8r\") pod \"e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") " pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.895741 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.895458    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"]
Oct 30 04:16:42.908103 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.905615    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:16:42.908103 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.905700    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:16:42.908103 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.905741    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:16:42.962863 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.962834    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:42.980202 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.980165    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7" event={"ID":"35fb8fc8-50a2-409b-9c88-8ad5219ace64","Type":"ContainerStarted","Data":"c3f7ec968483d8d7f1cb6bfd6cd13a41e82fb06d023e9779ab508858454edb20"}
Oct 30 04:16:42.980332 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.980233    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7" event={"ID":"35fb8fc8-50a2-409b-9c88-8ad5219ace64","Type":"ContainerStarted","Data":"e73e24a6c67f025e3b3bbc62341336728c33206b8bc3538589cdb9a3c90107c8"}
Oct 30 04:16:42.982671 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.982648    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb" event={"ID":"425e4e27-91f5-47b0-b8cf-311b5550e48a","Type":"ContainerStarted","Data":"d34846516b9345867589e5fdcf7f8e186eed965b05d45540178502c919f6f11f"}
Oct 30 04:16:42.982799 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:42.982687    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb" event={"ID":"425e4e27-91f5-47b0-b8cf-311b5550e48a","Type":"ContainerStarted","Data":"471821d323941e5cfe92e229fc3810316776c78f70a155ac71f95accc65b1a22"}
Oct 30 04:16:43.028841 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:43.026654    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7" podStartSLOduration=1.026633173 podStartE2EDuration="1.026633173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:43.022798844 +0000 UTC m=+1122.435194414" watchObservedRunningTime="2026-10-30 04:16:43.026633173 +0000 UTC m=+1122.439028746"
Oct 30 04:16:43.277822 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:43.277769    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb" podStartSLOduration=1.277752402 podStartE2EDuration="1.277752402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:43.068673837 +0000 UTC m=+1122.481069409" watchObservedRunningTime="2026-10-30 04:16:43.277752402 +0000 UTC m=+1122.690147977"
Oct 30 04:16:43.279555 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:43.279531    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"]
Oct 30 04:16:43.281003 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:16:43.280978    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a09d87c_a553_4a74_8559_59141d1778a4.slice/crio-5b59bf29dcc77850a270efb6ff4f6735389ce1d267f84df6b0efc84668a267ca WatchSource:0}: Error finding container 5b59bf29dcc77850a270efb6ff4f6735389ce1d267f84df6b0efc84668a267ca: Status 404 returned error can't find the container with id 5b59bf29dcc77850a270efb6ff4f6735389ce1d267f84df6b0efc84668a267ca
Oct 30 04:16:43.283358 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:43.283319    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:16:43.283439 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:43.283400    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:16:43.283481 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:43.283445    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:16:43.416333 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:43.416299    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:43.420883 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:43.420860    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:43.593063 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:43.592991    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:43.597415 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:43.597393    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:44.012263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:44.011380    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d" event={"ID":"1a09d87c-a553-4a74-8559-59141d1778a4","Type":"ContainerStarted","Data":"143a21015aba4d23c136bdb21e96d6e6ae261d35a94e4673bb0ee466137aa36c"}
Oct 30 04:16:44.012263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:44.011467    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d" event={"ID":"1a09d87c-a553-4a74-8559-59141d1778a4","Type":"ContainerStarted","Data":"5b59bf29dcc77850a270efb6ff4f6735389ce1d267f84df6b0efc84668a267ca"}
Oct 30 04:16:44.012263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:44.011497    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:44.026337 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:44.020396    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:44.026337 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:44.021765    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:16:44.026337 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:44.021938    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:16:44.055542 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:44.055487    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d" podStartSLOduration=2.055472844 podStartE2EDuration="2.055472844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:44.054454697 +0000 UTC m=+1123.466850252" watchObservedRunningTime="2026-10-30 04:16:44.055472844 +0000 UTC m=+1123.467868417"
Oct 30 04:16:44.964095 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:44.964061    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:44.968881 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:44.968856    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:45.015369 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:45.015336    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:45.016149 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:45.016130    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:16:46.185496 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:46.185467    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z"]
Oct 30 04:16:46.189006 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:46.188989    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z"
Oct 30 04:16:46.213954 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:46.213932    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-eb861bd9-serving-cert\""
Oct 30 04:16:46.213954 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:46.213946    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 04:16:46.214116 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:46.213990    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gw5bk\""
Oct 30 04:16:46.220526 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:46.220505    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z"]
Oct 30 04:16:46.301750 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:46.301714    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd756120-50b8-4deb-8488-9172698c606b-metrics-certs\") pod \"maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z\" (UID: \"cd756120-50b8-4deb-8488-9172698c606b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z"
Oct 30 04:16:46.301750 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:46.301753    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd756120-50b8-4deb-8488-9172698c606b-openshift-service-ca\") pod \"maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z\" (UID: \"cd756120-50b8-4deb-8488-9172698c606b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z"
Oct 30 04:16:46.301941 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:46.301851    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd756120-50b8-4deb-8488-9172698c606b-maas-api-tls\") pod \"maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z\" (UID: \"cd756120-50b8-4deb-8488-9172698c606b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z"
Oct 30 04:16:46.301941 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:46.301880    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmd9s\" (UniqueName: \"kubernetes.io/projected/cd756120-50b8-4deb-8488-9172698c606b-kube-api-access-wmd9s\") pod \"maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z\" (UID: \"cd756120-50b8-4deb-8488-9172698c606b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z"
Oct 30 04:16:46.402625 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:46.402594    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd756120-50b8-4deb-8488-9172698c606b-maas-api-tls\") pod \"maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z\" (UID: \"cd756120-50b8-4deb-8488-9172698c606b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z"
Oct 30 04:16:46.402790 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:46.402631    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmd9s\" (UniqueName: \"kubernetes.io/projected/cd756120-50b8-4deb-8488-9172698c606b-kube-api-access-wmd9s\") pod \"maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z\" (UID: \"cd756120-50b8-4deb-8488-9172698c606b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z"
Oct 30 04:16:46.402790 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:46.402689    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd756120-50b8-4deb-8488-9172698c606b-metrics-certs\") pod \"maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z\" (UID: \"cd756120-50b8-4deb-8488-9172698c606b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z"
Oct 30 04:16:46.402790 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:46.402709    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd756120-50b8-4deb-8488-9172698c606b-openshift-service-ca\") pod \"maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z\" (UID: \"cd756120-50b8-4deb-8488-9172698c606b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z"
Oct 30 04:16:46.403451 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:46.403427    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd756120-50b8-4deb-8488-9172698c606b-openshift-service-ca\") pod \"maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z\" (UID: \"cd756120-50b8-4deb-8488-9172698c606b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z"
Oct 30 04:16:46.405027 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:46.404998    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd756120-50b8-4deb-8488-9172698c606b-maas-api-tls\") pod \"maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z\" (UID: \"cd756120-50b8-4deb-8488-9172698c606b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z"
Oct 30 04:16:46.405124 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:46.405008    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd756120-50b8-4deb-8488-9172698c606b-metrics-certs\") pod \"maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z\" (UID: \"cd756120-50b8-4deb-8488-9172698c606b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z"
Oct 30 04:16:46.428723 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:46.428695    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmd9s\" (UniqueName: \"kubernetes.io/projected/cd756120-50b8-4deb-8488-9172698c606b-kube-api-access-wmd9s\") pod \"maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z\" (UID: \"cd756120-50b8-4deb-8488-9172698c606b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z"
Oct 30 04:16:46.500837 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:46.500802    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z"
Oct 30 04:16:46.801175 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:46.801151    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z"]
Oct 30 04:16:46.802075 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:16:46.802047    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd756120_50b8_4deb_8488_9172698c606b.slice/crio-6a9d36e11ec99fe5970ef4365c972e88f5e75e9999a62c753dc1c5ca458997bb WatchSource:0}: Error finding container 6a9d36e11ec99fe5970ef4365c972e88f5e75e9999a62c753dc1c5ca458997bb: Status 404 returned error can't find the container with id 6a9d36e11ec99fe5970ef4365c972e88f5e75e9999a62c753dc1c5ca458997bb
Oct 30 04:16:47.024413 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:47.024376    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z" event={"ID":"cd756120-50b8-4deb-8488-9172698c606b","Type":"ContainerStarted","Data":"6a9d36e11ec99fe5970ef4365c972e88f5e75e9999a62c753dc1c5ca458997bb"}
Oct 30 04:16:48.030036 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:48.029999    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z" event={"ID":"cd756120-50b8-4deb-8488-9172698c606b","Type":"ContainerStarted","Data":"8eab4bccb7cef8f06a4b7d1e228983de68469d70ad42775f8b53ae80cf740bd3"}
Oct 30 04:16:48.030457 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:48.030083    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z"
Oct 30 04:16:48.117902 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:48.112452    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z" podStartSLOduration=1.730244525 podStartE2EDuration="2.112430911s" totalImagesPullingTime="382.186386ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:48.111646933 +0000 UTC m=+1127.524042532" watchObservedRunningTime="2026-10-30 04:16:48.112430911 +0000 UTC m=+1127.524826485"
Oct 30 04:16:48.415575 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:48.415499    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn"]
Oct 30 04:16:48.419387 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:48.419367    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn"
Oct 30 04:16:48.433540 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:48.433510    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-566c3724-serving-cert\""
Oct 30 04:16:48.474023 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:48.473995    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn"]
Oct 30 04:16:48.530584 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:48.530552    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75b484fb-f875-402c-b213-2b86ad5a9889-maas-api-tls\") pod \"maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn\" (UID: \"75b484fb-f875-402c-b213-2b86ad5a9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn"
Oct 30 04:16:48.530705 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:48.530613    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75b484fb-f875-402c-b213-2b86ad5a9889-metrics-certs\") pod \"maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn\" (UID: \"75b484fb-f875-402c-b213-2b86ad5a9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn"
Oct 30 04:16:48.530705 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:48.530681    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vftwc\" (UniqueName: \"kubernetes.io/projected/75b484fb-f875-402c-b213-2b86ad5a9889-kube-api-access-vftwc\") pod \"maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn\" (UID: \"75b484fb-f875-402c-b213-2b86ad5a9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn"
Oct 30 04:16:48.530805 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:48.530719    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b484fb-f875-402c-b213-2b86ad5a9889-openshift-service-ca\") pod \"maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn\" (UID: \"75b484fb-f875-402c-b213-2b86ad5a9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn"
Oct 30 04:16:48.631230 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:48.631198    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75b484fb-f875-402c-b213-2b86ad5a9889-maas-api-tls\") pod \"maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn\" (UID: \"75b484fb-f875-402c-b213-2b86ad5a9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn"
Oct 30 04:16:48.631415 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:48.631367    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75b484fb-f875-402c-b213-2b86ad5a9889-metrics-certs\") pod \"maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn\" (UID: \"75b484fb-f875-402c-b213-2b86ad5a9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn"
Oct 30 04:16:48.631415 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:16:48.631383    4874 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-serving-cert: secret "maas-api-e2e-migrate-566c3724-serving-cert" not found
Oct 30 04:16:48.631415 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:48.631413    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vftwc\" (UniqueName: \"kubernetes.io/projected/75b484fb-f875-402c-b213-2b86ad5a9889-kube-api-access-vftwc\") pod \"maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn\" (UID: \"75b484fb-f875-402c-b213-2b86ad5a9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn"
Oct 30 04:16:48.631572 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:48.631444    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b484fb-f875-402c-b213-2b86ad5a9889-openshift-service-ca\") pod \"maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn\" (UID: \"75b484fb-f875-402c-b213-2b86ad5a9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn"
Oct 30 04:16:48.631572 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:16:48.631487    4874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b484fb-f875-402c-b213-2b86ad5a9889-maas-api-tls podName:75b484fb-f875-402c-b213-2b86ad5a9889 nodeName:}" failed. No retries permitted until 2026-10-30 04:16:49.131464656 +0000 UTC m=+1128.543860223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/75b484fb-f875-402c-b213-2b86ad5a9889-maas-api-tls") pod "maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn" (UID: "75b484fb-f875-402c-b213-2b86ad5a9889") : secret "maas-api-e2e-migrate-566c3724-serving-cert" not found
Oct 30 04:16:48.632080 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:48.632064    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b484fb-f875-402c-b213-2b86ad5a9889-openshift-service-ca\") pod \"maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn\" (UID: \"75b484fb-f875-402c-b213-2b86ad5a9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn"
Oct 30 04:16:48.633686 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:48.633666    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75b484fb-f875-402c-b213-2b86ad5a9889-metrics-certs\") pod \"maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn\" (UID: \"75b484fb-f875-402c-b213-2b86ad5a9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn"
Oct 30 04:16:48.667654 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:48.667591    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vftwc\" (UniqueName: \"kubernetes.io/projected/75b484fb-f875-402c-b213-2b86ad5a9889-kube-api-access-vftwc\") pod \"maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn\" (UID: \"75b484fb-f875-402c-b213-2b86ad5a9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn"
Oct 30 04:16:49.136141 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:49.136112    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75b484fb-f875-402c-b213-2b86ad5a9889-maas-api-tls\") pod \"maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn\" (UID: \"75b484fb-f875-402c-b213-2b86ad5a9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn"
Oct 30 04:16:49.138615 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:49.138586    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75b484fb-f875-402c-b213-2b86ad5a9889-maas-api-tls\") pod \"maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn\" (UID: \"75b484fb-f875-402c-b213-2b86ad5a9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn"
Oct 30 04:16:49.332486 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:49.332447    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn"
Oct 30 04:16:49.378112 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:49.378073    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:16:49.644119 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:49.644042    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn"]
Oct 30 04:16:49.647925 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:16:49.647891    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b484fb_f875_402c_b213_2b86ad5a9889.slice/crio-dbf043e7e72679eaabd7c878c7cafb26c296ae414f8fc2ad62dd62bef8ac9e57 WatchSource:0}: Error finding container dbf043e7e72679eaabd7c878c7cafb26c296ae414f8fc2ad62dd62bef8ac9e57: Status 404 returned error can't find the container with id dbf043e7e72679eaabd7c878c7cafb26c296ae414f8fc2ad62dd62bef8ac9e57
Oct 30 04:16:50.065410 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:50.065320    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn" event={"ID":"75b484fb-f875-402c-b213-2b86ad5a9889","Type":"ContainerStarted","Data":"dbf043e7e72679eaabd7c878c7cafb26c296ae414f8fc2ad62dd62bef8ac9e57"}
Oct 30 04:16:50.589331 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:50.587056    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7"]
Oct 30 04:16:50.592021 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:50.591996    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7"
Oct 30 04:16:50.614250 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:50.614213    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-de5733f9-serving-cert\""
Oct 30 04:16:50.628321 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:50.628296    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7"]
Oct 30 04:16:50.693121 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:50.693080    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/467160dd-4d32-401d-aea7-5692c748164f-openshift-service-ca\") pod \"maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7\" (UID: \"467160dd-4d32-401d-aea7-5692c748164f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7"
Oct 30 04:16:50.693314 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:50.693137    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/467160dd-4d32-401d-aea7-5692c748164f-maas-api-tls\") pod \"maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7\" (UID: \"467160dd-4d32-401d-aea7-5692c748164f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7"
Oct 30 04:16:50.693314 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:50.693253    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/467160dd-4d32-401d-aea7-5692c748164f-metrics-certs\") pod \"maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7\" (UID: \"467160dd-4d32-401d-aea7-5692c748164f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7"
Oct 30 04:16:50.693314 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:50.693300    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8kkx\" (UniqueName: \"kubernetes.io/projected/467160dd-4d32-401d-aea7-5692c748164f-kube-api-access-k8kkx\") pod \"maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7\" (UID: \"467160dd-4d32-401d-aea7-5692c748164f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7"
Oct 30 04:16:50.794861 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:50.794823    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/467160dd-4d32-401d-aea7-5692c748164f-openshift-service-ca\") pod \"maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7\" (UID: \"467160dd-4d32-401d-aea7-5692c748164f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7"
Oct 30 04:16:50.795064 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:50.794883    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/467160dd-4d32-401d-aea7-5692c748164f-maas-api-tls\") pod \"maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7\" (UID: \"467160dd-4d32-401d-aea7-5692c748164f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7"
Oct 30 04:16:50.795064 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:50.794958    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/467160dd-4d32-401d-aea7-5692c748164f-metrics-certs\") pod \"maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7\" (UID: \"467160dd-4d32-401d-aea7-5692c748164f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7"
Oct 30 04:16:50.795064 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:50.795004    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8kkx\" (UniqueName: \"kubernetes.io/projected/467160dd-4d32-401d-aea7-5692c748164f-kube-api-access-k8kkx\") pod \"maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7\" (UID: \"467160dd-4d32-401d-aea7-5692c748164f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7"
Oct 30 04:16:50.795697 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:50.795665    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/467160dd-4d32-401d-aea7-5692c748164f-openshift-service-ca\") pod \"maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7\" (UID: \"467160dd-4d32-401d-aea7-5692c748164f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7"
Oct 30 04:16:50.798226 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:50.798194    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/467160dd-4d32-401d-aea7-5692c748164f-metrics-certs\") pod \"maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7\" (UID: \"467160dd-4d32-401d-aea7-5692c748164f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7"
Oct 30 04:16:50.798359 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:50.798298    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/467160dd-4d32-401d-aea7-5692c748164f-maas-api-tls\") pod \"maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7\" (UID: \"467160dd-4d32-401d-aea7-5692c748164f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7"
Oct 30 04:16:50.830451 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:50.830392    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8kkx\" (UniqueName: \"kubernetes.io/projected/467160dd-4d32-401d-aea7-5692c748164f-kube-api-access-k8kkx\") pod \"maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7\" (UID: \"467160dd-4d32-401d-aea7-5692c748164f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7"
Oct 30 04:16:50.904821 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:50.904734    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7"
Oct 30 04:16:51.093035 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:51.093003    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn" event={"ID":"75b484fb-f875-402c-b213-2b86ad5a9889","Type":"ContainerStarted","Data":"25fd566b9c38162bf6b69bf194c92acef4fb932afc897f200987b75b2d67943a"}
Oct 30 04:16:51.093228 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:51.093118    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn"
Oct 30 04:16:51.134378 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:51.134328    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn" podStartSLOduration=2.7807828199999998 podStartE2EDuration="3.134314714s" totalImagesPullingTime="353.531894ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:51.133935868 +0000 UTC m=+1130.546331440" watchObservedRunningTime="2026-10-30 04:16:51.134314714 +0000 UTC m=+1130.546710287"
Oct 30 04:16:51.207377 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:51.207299    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7"]
Oct 30 04:16:51.209908 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:16:51.209885    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467160dd_4d32_401d_aea7_5692c748164f.slice/crio-77ad3c7f0ff4cbeccfdc841e9a66940e873ad414d58f4aaf69dd600da9855df3 WatchSource:0}: Error finding container 77ad3c7f0ff4cbeccfdc841e9a66940e873ad414d58f4aaf69dd600da9855df3: Status 404 returned error can't find the container with id 77ad3c7f0ff4cbeccfdc841e9a66940e873ad414d58f4aaf69dd600da9855df3
Oct 30 04:16:52.106388 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:52.106351    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7" event={"ID":"467160dd-4d32-401d-aea7-5692c748164f","Type":"ContainerStarted","Data":"61e8eee14c1657cea065874cec509a106dcba62a8172e16c3c709de828bde3e7"}
Oct 30 04:16:52.106822 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:52.106448    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7" event={"ID":"467160dd-4d32-401d-aea7-5692c748164f","Type":"ContainerStarted","Data":"77ad3c7f0ff4cbeccfdc841e9a66940e873ad414d58f4aaf69dd600da9855df3"}
Oct 30 04:16:52.106822 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:52.106510    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7"
Oct 30 04:16:52.159161 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:52.159099    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7" podStartSLOduration=1.7114434429999998 podStartE2EDuration="2.159082138s" totalImagesPullingTime="447.638695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:52.158483205 +0000 UTC m=+1131.570878779" watchObservedRunningTime="2026-10-30 04:16:52.159082138 +0000 UTC m=+1131.571477710"
Oct 30 04:16:54.040385 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:54.040359    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z"
Oct 30 04:16:54.265292 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:54.265255    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:16:54.381002 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:54.380909    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56"]
Oct 30 04:16:54.405750 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:54.405720    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56"]
Oct 30 04:16:54.406683 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:54.406607    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56"
Oct 30 04:16:54.450388 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:54.450359    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ad07bf73-serving-cert\""
Oct 30 04:16:54.452398 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:54.452371    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwvkq\" (UniqueName: \"kubernetes.io/projected/416f0a2e-bd70-499c-bafe-0b85913694c5-kube-api-access-jwvkq\") pod \"maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56\" (UID: \"416f0a2e-bd70-499c-bafe-0b85913694c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56"
Oct 30 04:16:54.452543 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:54.452413    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/416f0a2e-bd70-499c-bafe-0b85913694c5-metrics-certs\") pod \"maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56\" (UID: \"416f0a2e-bd70-499c-bafe-0b85913694c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56"
Oct 30 04:16:54.452543 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:54.452479    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416f0a2e-bd70-499c-bafe-0b85913694c5-maas-api-tls\") pod \"maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56\" (UID: \"416f0a2e-bd70-499c-bafe-0b85913694c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56"
Oct 30 04:16:54.452652 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:54.452587    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416f0a2e-bd70-499c-bafe-0b85913694c5-openshift-service-ca\") pod \"maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56\" (UID: \"416f0a2e-bd70-499c-bafe-0b85913694c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56"
Oct 30 04:16:54.553211 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:54.553179    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416f0a2e-bd70-499c-bafe-0b85913694c5-maas-api-tls\") pod \"maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56\" (UID: \"416f0a2e-bd70-499c-bafe-0b85913694c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56"
Oct 30 04:16:54.553373 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:54.553315    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416f0a2e-bd70-499c-bafe-0b85913694c5-openshift-service-ca\") pod \"maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56\" (UID: \"416f0a2e-bd70-499c-bafe-0b85913694c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56"
Oct 30 04:16:54.553429 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:54.553379    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwvkq\" (UniqueName: \"kubernetes.io/projected/416f0a2e-bd70-499c-bafe-0b85913694c5-kube-api-access-jwvkq\") pod \"maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56\" (UID: \"416f0a2e-bd70-499c-bafe-0b85913694c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56"
Oct 30 04:16:54.553429 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:54.553404    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/416f0a2e-bd70-499c-bafe-0b85913694c5-metrics-certs\") pod \"maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56\" (UID: \"416f0a2e-bd70-499c-bafe-0b85913694c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56"
Oct 30 04:16:54.554200 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:54.554163    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416f0a2e-bd70-499c-bafe-0b85913694c5-openshift-service-ca\") pod \"maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56\" (UID: \"416f0a2e-bd70-499c-bafe-0b85913694c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56"
Oct 30 04:16:54.555656 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:54.555629    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416f0a2e-bd70-499c-bafe-0b85913694c5-maas-api-tls\") pod \"maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56\" (UID: \"416f0a2e-bd70-499c-bafe-0b85913694c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56"
Oct 30 04:16:54.555818 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:54.555671    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/416f0a2e-bd70-499c-bafe-0b85913694c5-metrics-certs\") pod \"maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56\" (UID: \"416f0a2e-bd70-499c-bafe-0b85913694c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56"
Oct 30 04:16:54.600612 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:54.600582    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwvkq\" (UniqueName: \"kubernetes.io/projected/416f0a2e-bd70-499c-bafe-0b85913694c5-kube-api-access-jwvkq\") pod \"maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56\" (UID: \"416f0a2e-bd70-499c-bafe-0b85913694c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56"
Oct 30 04:16:54.759261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:54.759204    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56"
Oct 30 04:16:55.097212 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:55.097136    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56"]
Oct 30 04:16:55.100700 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:16:55.100611    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416f0a2e_bd70_499c_bafe_0b85913694c5.slice/crio-479ae8b1a846fc37ec68eecf84c6eb2e90bc2517f0c5b2f46eca05c381c51de8 WatchSource:0}: Error finding container 479ae8b1a846fc37ec68eecf84c6eb2e90bc2517f0c5b2f46eca05c381c51de8: Status 404 returned error can't find the container with id 479ae8b1a846fc37ec68eecf84c6eb2e90bc2517f0c5b2f46eca05c381c51de8
Oct 30 04:16:55.120120 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:55.120088    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56" event={"ID":"416f0a2e-bd70-499c-bafe-0b85913694c5","Type":"ContainerStarted","Data":"479ae8b1a846fc37ec68eecf84c6eb2e90bc2517f0c5b2f46eca05c381c51de8"}
Oct 30 04:16:55.734259 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:16:55.732497    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:16:56.128287 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:56.128179    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56" event={"ID":"416f0a2e-bd70-499c-bafe-0b85913694c5","Type":"ContainerStarted","Data":"4318fa7f9d22b6a73f5be01441abf7255715969f4debc3e9e5b3ea9c1ca83166"}
Oct 30 04:16:56.195262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:56.186968    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56" podStartSLOduration=1.827419905 podStartE2EDuration="2.186946577s" totalImagesPullingTime="359.526672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:56.185807532 +0000 UTC m=+1135.598203106" watchObservedRunningTime="2026-10-30 04:16:56.186946577 +0000 UTC m=+1135.599342151"
Oct 30 04:16:57.118176 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:57.118144    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn"
Oct 30 04:16:57.132995 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:57.132967    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56"
Oct 30 04:16:59.028272 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:59.027400    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:16:59.132736 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:16:59.132704    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7"
Oct 30 04:17:00.055408 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:00.055369    4874 patch_prober.go:29] interesting pod/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 04:17:00.055897 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:00.055436    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7" podUID="35fb8fc8-50a2-409b-9c88-8ad5219ace64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 04:17:02.165096 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:02.165062    4874 generic.go:356] "Generic (PLEG): container finished" podID="35fb8fc8-50a2-409b-9c88-8ad5219ace64" containerID="c3f7ec968483d8d7f1cb6bfd6cd13a41e82fb06d023e9779ab508858454edb20" exitCode=0
Oct 30 04:17:02.165096 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:02.165107    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7" event={"ID":"35fb8fc8-50a2-409b-9c88-8ad5219ace64","Type":"ContainerDied","Data":"c3f7ec968483d8d7f1cb6bfd6cd13a41e82fb06d023e9779ab508858454edb20"}
Oct 30 04:17:02.165595 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:02.165543    4874 scope.go:122] "RemoveContainer" containerID="c3f7ec968483d8d7f1cb6bfd6cd13a41e82fb06d023e9779ab508858454edb20"
Oct 30 04:17:02.166881 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:02.166848    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:02.166980 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:02.166920    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:02.166980 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:02.166953    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:03.144342 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:03.144314    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56"
Oct 30 04:17:03.207682 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:03.206597    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7" event={"ID":"35fb8fc8-50a2-409b-9c88-8ad5219ace64","Type":"ContainerStarted","Data":"d89c4371bf215aa07459c67a18c84b85062538d641f06aa80568a5b8f6108c70"}
Oct 30 04:17:03.596262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:03.578378    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h"]
Oct 30 04:17:03.596262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:03.594906    4874 patch_prober.go:29] interesting pod/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 30 04:17:03.596262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:03.594974    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7" podUID="35fb8fc8-50a2-409b-9c88-8ad5219ace64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 30 04:17:03.598580 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:03.597804    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:17:03.598580 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:03.597917    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h"
Oct 30 04:17:03.609277 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:03.609013    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h"]
Oct 30 04:17:03.621824 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:03.621792    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-cb1a18be-serving-cert\""
Oct 30 04:17:03.786264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:03.764498    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gmsc\" (UniqueName: \"kubernetes.io/projected/7001f753-082c-4759-b6e4-689d1d512d68-kube-api-access-7gmsc\") pod \"maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h\" (UID: \"7001f753-082c-4759-b6e4-689d1d512d68\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h"
Oct 30 04:17:03.786264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:03.764588    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7001f753-082c-4759-b6e4-689d1d512d68-maas-api-tls\") pod \"maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h\" (UID: \"7001f753-082c-4759-b6e4-689d1d512d68\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h"
Oct 30 04:17:03.786264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:03.764646    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7001f753-082c-4759-b6e4-689d1d512d68-metrics-certs\") pod \"maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h\" (UID: \"7001f753-082c-4759-b6e4-689d1d512d68\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h"
Oct 30 04:17:03.786264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:03.764735    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7001f753-082c-4759-b6e4-689d1d512d68-openshift-service-ca\") pod \"maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h\" (UID: \"7001f753-082c-4759-b6e4-689d1d512d68\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h"
Oct 30 04:17:03.867849 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:03.866789    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7001f753-082c-4759-b6e4-689d1d512d68-maas-api-tls\") pod \"maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h\" (UID: \"7001f753-082c-4759-b6e4-689d1d512d68\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h"
Oct 30 04:17:03.867849 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:03.866868    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7001f753-082c-4759-b6e4-689d1d512d68-metrics-certs\") pod \"maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h\" (UID: \"7001f753-082c-4759-b6e4-689d1d512d68\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h"
Oct 30 04:17:03.867849 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:03.866954    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7001f753-082c-4759-b6e4-689d1d512d68-openshift-service-ca\") pod \"maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h\" (UID: \"7001f753-082c-4759-b6e4-689d1d512d68\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h"
Oct 30 04:17:03.867849 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:03.867003    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gmsc\" (UniqueName: \"kubernetes.io/projected/7001f753-082c-4759-b6e4-689d1d512d68-kube-api-access-7gmsc\") pod \"maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h\" (UID: \"7001f753-082c-4759-b6e4-689d1d512d68\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h"
Oct 30 04:17:03.868954 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:03.868892    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7001f753-082c-4759-b6e4-689d1d512d68-openshift-service-ca\") pod \"maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h\" (UID: \"7001f753-082c-4759-b6e4-689d1d512d68\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h"
Oct 30 04:17:03.900692 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:03.900666    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7001f753-082c-4759-b6e4-689d1d512d68-metrics-certs\") pod \"maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h\" (UID: \"7001f753-082c-4759-b6e4-689d1d512d68\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h"
Oct 30 04:17:03.900860 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:03.900838    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7001f753-082c-4759-b6e4-689d1d512d68-maas-api-tls\") pod \"maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h\" (UID: \"7001f753-082c-4759-b6e4-689d1d512d68\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h"
Oct 30 04:17:03.902079 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:03.902055    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gmsc\" (UniqueName: \"kubernetes.io/projected/7001f753-082c-4759-b6e4-689d1d512d68-kube-api-access-7gmsc\") pod \"maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h\" (UID: \"7001f753-082c-4759-b6e4-689d1d512d68\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h"
Oct 30 04:17:03.922807 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:03.922776    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h"
Oct 30 04:17:04.263364 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:04.263338    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h"]
Oct 30 04:17:04.593889 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:04.593855    4874 patch_prober.go:29] interesting pod/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 30 04:17:04.594057 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:04.593920    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7" podUID="35fb8fc8-50a2-409b-9c88-8ad5219ace64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 30 04:17:05.218318 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:05.218179    4874 generic.go:356] "Generic (PLEG): container finished" podID="1a09d87c-a553-4a74-8559-59141d1778a4" containerID="143a21015aba4d23c136bdb21e96d6e6ae261d35a94e4673bb0ee466137aa36c" exitCode=0
Oct 30 04:17:05.218318 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:05.218271    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d" event={"ID":"1a09d87c-a553-4a74-8559-59141d1778a4","Type":"ContainerDied","Data":"143a21015aba4d23c136bdb21e96d6e6ae261d35a94e4673bb0ee466137aa36c"}
Oct 30 04:17:05.218718 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:05.218697    4874 scope.go:122] "RemoveContainer" containerID="143a21015aba4d23c136bdb21e96d6e6ae261d35a94e4673bb0ee466137aa36c"
Oct 30 04:17:05.220155 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:05.220129    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:05.220224 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:05.220194    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:05.220280 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:05.220226    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:05.220280 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:05.220248    4874 generic.go:356] "Generic (PLEG): container finished" podID="425e4e27-91f5-47b0-b8cf-311b5550e48a" containerID="d34846516b9345867589e5fdcf7f8e186eed965b05d45540178502c919f6f11f" exitCode=0
Oct 30 04:17:05.220354 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:05.220311    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb" event={"ID":"425e4e27-91f5-47b0-b8cf-311b5550e48a","Type":"ContainerDied","Data":"d34846516b9345867589e5fdcf7f8e186eed965b05d45540178502c919f6f11f"}
Oct 30 04:17:05.220673 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:05.220599    4874 scope.go:122] "RemoveContainer" containerID="d34846516b9345867589e5fdcf7f8e186eed965b05d45540178502c919f6f11f"
Oct 30 04:17:05.221730 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:05.221692    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:05.221818 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:05.221783    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:05.221881 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:05.221830    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:05.222375 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:05.222320    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h" event={"ID":"7001f753-082c-4759-b6e4-689d1d512d68","Type":"ContainerStarted","Data":"90236b2a5e7952c4e111f31fedd0c919eeff2d076771066825574283e167675e"}
Oct 30 04:17:05.222375 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:05.222352    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h" event={"ID":"7001f753-082c-4759-b6e4-689d1d512d68","Type":"ContainerStarted","Data":"5398cedbfab5e913de6df58d3733c89a38dada5975ca8cfad7f6d08aa22f75a2"}
Oct 30 04:17:05.222548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:05.222482    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h"
Oct 30 04:17:05.387262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:05.380102    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h" podStartSLOduration=1.956258115 podStartE2EDuration="2.380081586s" totalImagesPullingTime="423.823471ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:05.379012251 +0000 UTC m=+1144.791407824" watchObservedRunningTime="2026-10-30 04:17:05.380081586 +0000 UTC m=+1144.792477161"
Oct 30 04:17:05.604263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:05.596745    4874 patch_prober.go:29] interesting pod/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 30 04:17:05.604263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:05.596809    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7" podUID="35fb8fc8-50a2-409b-9c88-8ad5219ace64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 30 04:17:06.245212 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:06.244051    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d" event={"ID":"1a09d87c-a553-4a74-8559-59141d1778a4","Type":"ContainerStarted","Data":"4580303c965b08c3cf86113f4b99df03b1dd9503da30078554a994abc5a88958"}
Oct 30 04:17:06.246980 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:06.246949    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb" event={"ID":"425e4e27-91f5-47b0-b8cf-311b5550e48a","Type":"ContainerStarted","Data":"739fab602a02e6c6af1d1e23b5a9826a98f02b427e58769a710c99ca5ba7dfae"}
Oct 30 04:17:06.420264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:06.419670    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:17:06.420264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:06.419800    4874 patch_prober.go:29] interesting pod/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Oct 30 04:17:06.420264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:06.419844    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb" podUID="425e4e27-91f5-47b0-b8cf-311b5550e48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Oct 30 04:17:06.594186 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:06.593681    4874 patch_prober.go:29] interesting pod/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 30 04:17:06.594186 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:06.593745    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7" podUID="35fb8fc8-50a2-409b-9c88-8ad5219ace64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 30 04:17:06.963617 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:06.963527    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:17:06.963799 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:06.963777    4874 patch_prober.go:29] interesting pod/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 04:17:06.963845 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:06.963824    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d" podUID="1a09d87c-a553-4a74-8559-59141d1778a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 04:17:07.416701 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:07.416662    4874 patch_prober.go:29] interesting pod/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Oct 30 04:17:07.416971 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:07.416941    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb" podUID="425e4e27-91f5-47b0-b8cf-311b5550e48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Oct 30 04:17:07.597264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:07.596657    4874 patch_prober.go:29] interesting pod/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 30 04:17:07.597264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:07.596721    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7" podUID="35fb8fc8-50a2-409b-9c88-8ad5219ace64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 30 04:17:07.964087 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:07.964045    4874 patch_prober.go:29] interesting pod/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 04:17:07.964408 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:07.964364    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d" podUID="1a09d87c-a553-4a74-8559-59141d1778a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 04:17:08.416758 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:08.416716    4874 patch_prober.go:29] interesting pod/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Oct 30 04:17:08.417037 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:08.417004    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb" podUID="425e4e27-91f5-47b0-b8cf-311b5550e48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Oct 30 04:17:08.436202 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:08.436162    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:17:08.626096 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:08.626054    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:17:08.626909 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:08.626882    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:17:08.628261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:08.627898    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:17:08.964055 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:08.964024    4874 patch_prober.go:29] interesting pod/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 04:17:08.964215 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:08.964075    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d" podUID="1a09d87c-a553-4a74-8559-59141d1778a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 04:17:09.416692 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:09.416654    4874 patch_prober.go:29] interesting pod/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Oct 30 04:17:09.416970 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:09.416940    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb" podUID="425e4e27-91f5-47b0-b8cf-311b5550e48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Oct 30 04:17:09.963666 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:09.963633    4874 patch_prober.go:29] interesting pod/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 04:17:09.964073 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:09.963683    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d" podUID="1a09d87c-a553-4a74-8559-59141d1778a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 04:17:10.416716 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:10.416684    4874 patch_prober.go:29] interesting pod/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Oct 30 04:17:10.416985 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:10.416963    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb" podUID="425e4e27-91f5-47b0-b8cf-311b5550e48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Oct 30 04:17:10.963734 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:10.963695    4874 patch_prober.go:29] interesting pod/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 04:17:10.964269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:10.963756    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d" podUID="1a09d87c-a553-4a74-8559-59141d1778a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 04:17:11.252797 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:11.252764    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h"
Oct 30 04:17:11.416802 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:11.416762    4874 patch_prober.go:29] interesting pod/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Oct 30 04:17:11.417006 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:11.416829    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb" podUID="425e4e27-91f5-47b0-b8cf-311b5550e48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Oct 30 04:17:11.967962 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:11.967928    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:17:11.968399 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:11.968253    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:17:11.968855 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:11.968838    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:17:12.420497 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:12.420468    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:17:12.420764 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:12.420744    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:17:12.421459 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:12.421440    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:17:18.101944 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.101886    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z"]
Oct 30 04:17:18.102544 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.102198    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z" podUID="cd756120-50b8-4deb-8488-9172698c606b" containerName="maas-api" containerID="cri-o://8eab4bccb7cef8f06a4b7d1e228983de68469d70ad42775f8b53ae80cf740bd3" gracePeriod=30
Oct 30 04:17:18.362545 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.362450    4874 generic.go:356] "Generic (PLEG): container finished" podID="cd756120-50b8-4deb-8488-9172698c606b" containerID="8eab4bccb7cef8f06a4b7d1e228983de68469d70ad42775f8b53ae80cf740bd3" exitCode=0
Oct 30 04:17:18.362545 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.362510    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z" event={"ID":"cd756120-50b8-4deb-8488-9172698c606b","Type":"ContainerDied","Data":"8eab4bccb7cef8f06a4b7d1e228983de68469d70ad42775f8b53ae80cf740bd3"}
Oct 30 04:17:18.362754 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.362558    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z" event={"ID":"cd756120-50b8-4deb-8488-9172698c606b","Type":"ContainerDied","Data":"6a9d36e11ec99fe5970ef4365c972e88f5e75e9999a62c753dc1c5ca458997bb"}
Oct 30 04:17:18.362754 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.362581    4874 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a9d36e11ec99fe5970ef4365c972e88f5e75e9999a62c753dc1c5ca458997bb"
Oct 30 04:17:18.367206 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.366908    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z"
Oct 30 04:17:18.464113 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.464083    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn"]
Oct 30 04:17:18.464336 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.464292    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn" podUID="75b484fb-f875-402c-b213-2b86ad5a9889" containerName="maas-api" containerID="cri-o://25fd566b9c38162bf6b69bf194c92acef4fb932afc897f200987b75b2d67943a" gracePeriod=30
Oct 30 04:17:18.465539 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.465514    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd756120-50b8-4deb-8488-9172698c606b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd756120-50b8-4deb-8488-9172698c606b-maas-api-tls\") pod \"cd756120-50b8-4deb-8488-9172698c606b\" (UID: \"cd756120-50b8-4deb-8488-9172698c606b\") "
Oct 30 04:17:18.465642 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.465619    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd756120-50b8-4deb-8488-9172698c606b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd756120-50b8-4deb-8488-9172698c606b-metrics-certs\") pod \"cd756120-50b8-4deb-8488-9172698c606b\" (UID: \"cd756120-50b8-4deb-8488-9172698c606b\") "
Oct 30 04:17:18.465705 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.465675    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd756120-50b8-4deb-8488-9172698c606b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd756120-50b8-4deb-8488-9172698c606b-openshift-service-ca\") pod \"cd756120-50b8-4deb-8488-9172698c606b\" (UID: \"cd756120-50b8-4deb-8488-9172698c606b\") "
Oct 30 04:17:18.465903 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.465865    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd756120-50b8-4deb-8488-9172698c606b-kube-api-access-wmd9s\" (UniqueName: \"kubernetes.io/projected/cd756120-50b8-4deb-8488-9172698c606b-kube-api-access-wmd9s\") pod \"cd756120-50b8-4deb-8488-9172698c606b\" (UID: \"cd756120-50b8-4deb-8488-9172698c606b\") "
Oct 30 04:17:18.466828 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.466755    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd756120-50b8-4deb-8488-9172698c606b-openshift-service-ca" pod "cd756120-50b8-4deb-8488-9172698c606b" (UID: "cd756120-50b8-4deb-8488-9172698c606b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:18.474112 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.474072    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd756120-50b8-4deb-8488-9172698c606b-metrics-certs" pod "cd756120-50b8-4deb-8488-9172698c606b" (UID: "cd756120-50b8-4deb-8488-9172698c606b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:18.477036 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.476769    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd756120-50b8-4deb-8488-9172698c606b-maas-api-tls" pod "cd756120-50b8-4deb-8488-9172698c606b" (UID: "cd756120-50b8-4deb-8488-9172698c606b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:18.478637 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.478611    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd756120-50b8-4deb-8488-9172698c606b-kube-api-access-wmd9s" pod "cd756120-50b8-4deb-8488-9172698c606b" (UID: "cd756120-50b8-4deb-8488-9172698c606b"). InnerVolumeSpecName "kube-api-access-wmd9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:18.567959 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.567928    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd756120-50b8-4deb-8488-9172698c606b-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:18.567959 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.567958    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd756120-50b8-4deb-8488-9172698c606b-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:18.568150 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.567972    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmd9s\" (UniqueName: \"kubernetes.io/projected/cd756120-50b8-4deb-8488-9172698c606b-kube-api-access-wmd9s\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:18.568150 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.567983    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd756120-50b8-4deb-8488-9172698c606b-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:18.754049 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.754025    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn"
Oct 30 04:17:18.831046 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.831009    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7"]
Oct 30 04:17:18.831299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.831273    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7" podUID="467160dd-4d32-401d-aea7-5692c748164f" containerName="maas-api" containerID="cri-o://61e8eee14c1657cea065874cec509a106dcba62a8172e16c3c709de828bde3e7" gracePeriod=30
Oct 30 04:17:18.873462 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.873430    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75b484fb-f875-402c-b213-2b86ad5a9889-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75b484fb-f875-402c-b213-2b86ad5a9889-maas-api-tls\") pod \"75b484fb-f875-402c-b213-2b86ad5a9889\" (UID: \"75b484fb-f875-402c-b213-2b86ad5a9889\") "
Oct 30 04:17:18.873615 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.873494    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b484fb-f875-402c-b213-2b86ad5a9889-kube-api-access-vftwc\" (UniqueName: \"kubernetes.io/projected/75b484fb-f875-402c-b213-2b86ad5a9889-kube-api-access-vftwc\") pod \"75b484fb-f875-402c-b213-2b86ad5a9889\" (UID: \"75b484fb-f875-402c-b213-2b86ad5a9889\") "
Oct 30 04:17:18.873615 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.873550    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75b484fb-f875-402c-b213-2b86ad5a9889-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b484fb-f875-402c-b213-2b86ad5a9889-openshift-service-ca\") pod \"75b484fb-f875-402c-b213-2b86ad5a9889\" (UID: \"75b484fb-f875-402c-b213-2b86ad5a9889\") "
Oct 30 04:17:18.873725 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.873696    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75b484fb-f875-402c-b213-2b86ad5a9889-metrics-certs\" (UniqueName: \"kubernetes.io/secret/75b484fb-f875-402c-b213-2b86ad5a9889-metrics-certs\") pod \"75b484fb-f875-402c-b213-2b86ad5a9889\" (UID: \"75b484fb-f875-402c-b213-2b86ad5a9889\") "
Oct 30 04:17:18.874041 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.874009    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b484fb-f875-402c-b213-2b86ad5a9889-openshift-service-ca" pod "75b484fb-f875-402c-b213-2b86ad5a9889" (UID: "75b484fb-f875-402c-b213-2b86ad5a9889"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:18.875708 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.875680    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75b484fb-f875-402c-b213-2b86ad5a9889-maas-api-tls" pod "75b484fb-f875-402c-b213-2b86ad5a9889" (UID: "75b484fb-f875-402c-b213-2b86ad5a9889"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:18.875794 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.875706    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75b484fb-f875-402c-b213-2b86ad5a9889-metrics-certs" pod "75b484fb-f875-402c-b213-2b86ad5a9889" (UID: "75b484fb-f875-402c-b213-2b86ad5a9889"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:18.875855 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.875838    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b484fb-f875-402c-b213-2b86ad5a9889-kube-api-access-vftwc" pod "75b484fb-f875-402c-b213-2b86ad5a9889" (UID: "75b484fb-f875-402c-b213-2b86ad5a9889"). InnerVolumeSpecName "kube-api-access-vftwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:18.931717 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.931642    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"]
Oct 30 04:17:18.931917 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.931892    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7" podUID="35fb8fc8-50a2-409b-9c88-8ad5219ace64" containerName="istio-proxy" containerID="cri-o://d89c4371bf215aa07459c67a18c84b85062538d641f06aa80568a5b8f6108c70" gracePeriod=30
Oct 30 04:17:18.975122 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.975087    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75b484fb-f875-402c-b213-2b86ad5a9889-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:18.975122 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.975125    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75b484fb-f875-402c-b213-2b86ad5a9889-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:18.975380 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.975140    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vftwc\" (UniqueName: \"kubernetes.io/projected/75b484fb-f875-402c-b213-2b86ad5a9889-kube-api-access-vftwc\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:18.975380 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:18.975155    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b484fb-f875-402c-b213-2b86ad5a9889-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:19.067439 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.067414    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7"
Oct 30 04:17:19.177112 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.177083    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/467160dd-4d32-401d-aea7-5692c748164f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/467160dd-4d32-401d-aea7-5692c748164f-openshift-service-ca\") pod \"467160dd-4d32-401d-aea7-5692c748164f\" (UID: \"467160dd-4d32-401d-aea7-5692c748164f\") "
Oct 30 04:17:19.177471 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.177172    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467160dd-4d32-401d-aea7-5692c748164f-kube-api-access-k8kkx\" (UniqueName: \"kubernetes.io/projected/467160dd-4d32-401d-aea7-5692c748164f-kube-api-access-k8kkx\") pod \"467160dd-4d32-401d-aea7-5692c748164f\" (UID: \"467160dd-4d32-401d-aea7-5692c748164f\") "
Oct 30 04:17:19.177471 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.177208    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/467160dd-4d32-401d-aea7-5692c748164f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/467160dd-4d32-401d-aea7-5692c748164f-maas-api-tls\") pod \"467160dd-4d32-401d-aea7-5692c748164f\" (UID: \"467160dd-4d32-401d-aea7-5692c748164f\") "
Oct 30 04:17:19.177471 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.177235    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/467160dd-4d32-401d-aea7-5692c748164f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/467160dd-4d32-401d-aea7-5692c748164f-metrics-certs\") pod \"467160dd-4d32-401d-aea7-5692c748164f\" (UID: \"467160dd-4d32-401d-aea7-5692c748164f\") "
Oct 30 04:17:19.177566 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.177499    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/467160dd-4d32-401d-aea7-5692c748164f-openshift-service-ca" pod "467160dd-4d32-401d-aea7-5692c748164f" (UID: "467160dd-4d32-401d-aea7-5692c748164f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:19.177610 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.177573    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/467160dd-4d32-401d-aea7-5692c748164f-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:19.179378 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.179345    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/467160dd-4d32-401d-aea7-5692c748164f-metrics-certs" pod "467160dd-4d32-401d-aea7-5692c748164f" (UID: "467160dd-4d32-401d-aea7-5692c748164f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:19.179488 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.179386    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/467160dd-4d32-401d-aea7-5692c748164f-maas-api-tls" pod "467160dd-4d32-401d-aea7-5692c748164f" (UID: "467160dd-4d32-401d-aea7-5692c748164f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:19.179488 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.179431    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467160dd-4d32-401d-aea7-5692c748164f-kube-api-access-k8kkx" pod "467160dd-4d32-401d-aea7-5692c748164f" (UID: "467160dd-4d32-401d-aea7-5692c748164f"). InnerVolumeSpecName "kube-api-access-k8kkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:19.278489 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.278320    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8kkx\" (UniqueName: \"kubernetes.io/projected/467160dd-4d32-401d-aea7-5692c748164f-kube-api-access-k8kkx\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:19.278489 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.278364    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/467160dd-4d32-401d-aea7-5692c748164f-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:19.278489 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.278378    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/467160dd-4d32-401d-aea7-5692c748164f-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:19.374108 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.370742    4874 generic.go:356] "Generic (PLEG): container finished" podID="75b484fb-f875-402c-b213-2b86ad5a9889" containerID="25fd566b9c38162bf6b69bf194c92acef4fb932afc897f200987b75b2d67943a" exitCode=0
Oct 30 04:17:19.374108 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.370804    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn"
Oct 30 04:17:19.374108 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.370808    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn" event={"ID":"75b484fb-f875-402c-b213-2b86ad5a9889","Type":"ContainerDied","Data":"25fd566b9c38162bf6b69bf194c92acef4fb932afc897f200987b75b2d67943a"}
Oct 30 04:17:19.374108 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.370873    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn" event={"ID":"75b484fb-f875-402c-b213-2b86ad5a9889","Type":"ContainerDied","Data":"dbf043e7e72679eaabd7c878c7cafb26c296ae414f8fc2ad62dd62bef8ac9e57"}
Oct 30 04:17:19.374108 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.370907    4874 scope.go:122] "RemoveContainer" containerID="25fd566b9c38162bf6b69bf194c92acef4fb932afc897f200987b75b2d67943a"
Oct 30 04:17:19.374108 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.370997    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"]
Oct 30 04:17:19.374108 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.371422    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d" podUID="1a09d87c-a553-4a74-8559-59141d1778a4" containerName="istio-proxy" containerID="cri-o://4580303c965b08c3cf86113f4b99df03b1dd9503da30078554a994abc5a88958" gracePeriod=30
Oct 30 04:17:19.374108 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.373316    4874 generic.go:356] "Generic (PLEG): container finished" podID="467160dd-4d32-401d-aea7-5692c748164f" containerID="61e8eee14c1657cea065874cec509a106dcba62a8172e16c3c709de828bde3e7" exitCode=0
Oct 30 04:17:19.374108 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.373385    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z"
Oct 30 04:17:19.374108 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.373416    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7" event={"ID":"467160dd-4d32-401d-aea7-5692c748164f","Type":"ContainerDied","Data":"61e8eee14c1657cea065874cec509a106dcba62a8172e16c3c709de828bde3e7"}
Oct 30 04:17:19.374108 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.373494    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7"
Oct 30 04:17:19.374108 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.373467    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7" event={"ID":"467160dd-4d32-401d-aea7-5692c748164f","Type":"ContainerDied","Data":"77ad3c7f0ff4cbeccfdc841e9a66940e873ad414d58f4aaf69dd600da9855df3"}
Oct 30 04:17:19.391819 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.391794    4874 scope.go:122] "RemoveContainer" containerID="25fd566b9c38162bf6b69bf194c92acef4fb932afc897f200987b75b2d67943a"
Oct 30 04:17:19.392118 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:17:19.392097    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25fd566b9c38162bf6b69bf194c92acef4fb932afc897f200987b75b2d67943a\": container with ID starting with 25fd566b9c38162bf6b69bf194c92acef4fb932afc897f200987b75b2d67943a not found: ID does not exist" containerID="25fd566b9c38162bf6b69bf194c92acef4fb932afc897f200987b75b2d67943a"
Oct 30 04:17:19.392180 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.392125    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25fd566b9c38162bf6b69bf194c92acef4fb932afc897f200987b75b2d67943a"} err="failed to get container status \"25fd566b9c38162bf6b69bf194c92acef4fb932afc897f200987b75b2d67943a\": rpc error: code = NotFound desc = could not find container \"25fd566b9c38162bf6b69bf194c92acef4fb932afc897f200987b75b2d67943a\": container with ID starting with 25fd566b9c38162bf6b69bf194c92acef4fb932afc897f200987b75b2d67943a not found: ID does not exist"
Oct 30 04:17:19.392180 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.392140    4874 scope.go:122] "RemoveContainer" containerID="61e8eee14c1657cea065874cec509a106dcba62a8172e16c3c709de828bde3e7"
Oct 30 04:17:19.400712 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.400697    4874 scope.go:122] "RemoveContainer" containerID="61e8eee14c1657cea065874cec509a106dcba62a8172e16c3c709de828bde3e7"
Oct 30 04:17:19.400953 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:17:19.400935    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61e8eee14c1657cea065874cec509a106dcba62a8172e16c3c709de828bde3e7\": container with ID starting with 61e8eee14c1657cea065874cec509a106dcba62a8172e16c3c709de828bde3e7 not found: ID does not exist" containerID="61e8eee14c1657cea065874cec509a106dcba62a8172e16c3c709de828bde3e7"
Oct 30 04:17:19.400994 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.400960    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61e8eee14c1657cea065874cec509a106dcba62a8172e16c3c709de828bde3e7"} err="failed to get container status \"61e8eee14c1657cea065874cec509a106dcba62a8172e16c3c709de828bde3e7\": rpc error: code = NotFound desc = could not find container \"61e8eee14c1657cea065874cec509a106dcba62a8172e16c3c709de828bde3e7\": container with ID starting with 61e8eee14c1657cea065874cec509a106dcba62a8172e16c3c709de828bde3e7 not found: ID does not exist"
Oct 30 04:17:19.501962 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.501905    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn"]
Oct 30 04:17:19.509732 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.507632    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-566c3724-569b9ff756-sm6vn"]
Oct 30 04:17:19.538929 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.538897    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z"]
Oct 30 04:17:19.543071 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.543047    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eb861bd9-5fd78c674-k7t9z"]
Oct 30 04:17:19.575812 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.575779    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7"]
Oct 30 04:17:19.581898 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:19.581874    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-de5733f9-755f4dcd47-vvlq7"]
Oct 30 04:17:20.248357 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.248326    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"]
Oct 30 04:17:20.248826 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.248745    4874 cpu_manager.go:414] "containerMap: removing container" podUID="75b484fb-f875-402c-b213-2b86ad5a9889" containerName="maas-api"
Oct 30 04:17:20.248826 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.248757    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b484fb-f875-402c-b213-2b86ad5a9889" containerName="maas-api"
Oct 30 04:17:20.248826 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.248771    4874 cpu_manager.go:414] "containerMap: removing container" podUID="467160dd-4d32-401d-aea7-5692c748164f" containerName="maas-api"
Oct 30 04:17:20.248826 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.248776    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467160dd-4d32-401d-aea7-5692c748164f" containerName="maas-api"
Oct 30 04:17:20.248826 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.248786    4874 cpu_manager.go:414] "containerMap: removing container" podUID="cd756120-50b8-4deb-8488-9172698c606b" containerName="maas-api"
Oct 30 04:17:20.248826 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.248791    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd756120-50b8-4deb-8488-9172698c606b" containerName="maas-api"
Oct 30 04:17:20.249122 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.248853    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="467160dd-4d32-401d-aea7-5692c748164f" containerName="maas-api"
Oct 30 04:17:20.249122 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.248863    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd756120-50b8-4deb-8488-9172698c606b" containerName="maas-api"
Oct 30 04:17:20.249122 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.248873    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b484fb-f875-402c-b213-2b86ad5a9889" containerName="maas-api"
Oct 30 04:17:20.251047 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.251007    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.269048 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.269028    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-5d74b9-openshift-default-dockercfg-d8777\""
Oct 30 04:17:20.281396 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.281375    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"]
Oct 30 04:17:20.291654 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.291629    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b28e83-cdc5-4591-858b-80df98885d4e-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.291784 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.291670    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b28e83-cdc5-4591-858b-80df98885d4e-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.291784 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.291687    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b28e83-cdc5-4591-858b-80df98885d4e-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.291784 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.291727    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.291957 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.291783    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.291957 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.291815    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.291957 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.291832    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.292105 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.291973    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.292105 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.292010    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjpd7\" (UniqueName: \"kubernetes.io/projected/10b28e83-cdc5-4591-858b-80df98885d4e-kube-api-access-mjpd7\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.393513 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.393479    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.393681 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.393529    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.393681 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.393633    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.393777 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.393686    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjpd7\" (UniqueName: \"kubernetes.io/projected/10b28e83-cdc5-4591-858b-80df98885d4e-kube-api-access-mjpd7\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.393777 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.393719    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b28e83-cdc5-4591-858b-80df98885d4e-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.393777 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.393761    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b28e83-cdc5-4591-858b-80df98885d4e-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.393878 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.393785    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b28e83-cdc5-4591-858b-80df98885d4e-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.393878 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.393814    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.393878 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.393850    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.394347 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.394319    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.396036 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.395972    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.397477 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.396835    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.397477 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.397041    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.398547 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.397733    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b28e83-cdc5-4591-858b-80df98885d4e-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.400819 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.400796    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.407411 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.406973    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b28e83-cdc5-4591-858b-80df98885d4e-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.418848 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.418821    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b28e83-cdc5-4591-858b-80df98885d4e-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.418982 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.418961    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjpd7\" (UniqueName: \"kubernetes.io/projected/10b28e83-cdc5-4591-858b-80df98885d4e-kube-api-access-mjpd7\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.565808 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.565736    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:20.694325 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.694202    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"]
Oct 30 04:17:20.697770 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.697748    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.717609 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.717586    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b4708a-openshift-default-dockercfg-t5rn6\""
Oct 30 04:17:20.727949 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.727921    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="467160dd-4d32-401d-aea7-5692c748164f" path="/var/lib/kubelet/pods/467160dd-4d32-401d-aea7-5692c748164f/volumes"
Oct 30 04:17:20.728534 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.728516    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75b484fb-f875-402c-b213-2b86ad5a9889" path="/var/lib/kubelet/pods/75b484fb-f875-402c-b213-2b86ad5a9889/volumes"
Oct 30 04:17:20.729041 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.729023    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd756120-50b8-4deb-8488-9172698c606b" path="/var/lib/kubelet/pods/cd756120-50b8-4deb-8488-9172698c606b/volumes"
Oct 30 04:17:20.729395 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.729379    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"]
Oct 30 04:17:20.797549 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.797518    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.797549 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.797551    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e716825-de92-4c9d-9b36-007ea4218a98-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.797789 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.797591    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.797789 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.797704    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e716825-de92-4c9d-9b36-007ea4218a98-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.797789 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.797735    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.797926 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.797910    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.798000 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.797978    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.798042 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.798018    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9whx\" (UniqueName: \"kubernetes.io/projected/4e716825-de92-4c9d-9b36-007ea4218a98-kube-api-access-t9whx\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.798079 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.798067    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e716825-de92-4c9d-9b36-007ea4218a98-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.887419 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.887287    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"]
Oct 30 04:17:20.892308 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:17:20.892279    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b28e83_cdc5_4591_858b_80df98885d4e.slice/crio-ad73c7d3a82c333a7fec58bfed95f090878db2f692b8514459abc2e9d650b57a WatchSource:0}: Error finding container ad73c7d3a82c333a7fec58bfed95f090878db2f692b8514459abc2e9d650b57a: Status 404 returned error can't find the container with id ad73c7d3a82c333a7fec58bfed95f090878db2f692b8514459abc2e9d650b57a
Oct 30 04:17:20.895150 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.895111    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:20.895256 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.895200    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:20.895300 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.895270    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:20.899680 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.899577    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e716825-de92-4c9d-9b36-007ea4218a98-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.899680 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.899642    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.900085 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.900007    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.900414 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.900055    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.900414 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.900207    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9whx\" (UniqueName: \"kubernetes.io/projected/4e716825-de92-4c9d-9b36-007ea4218a98-kube-api-access-t9whx\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.900639 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.900612    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e716825-de92-4c9d-9b36-007ea4218a98-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.900760 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.900741    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.900820 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.900774    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e716825-de92-4c9d-9b36-007ea4218a98-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.900875 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.900825    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.903335 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.902342    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.903335 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.902832    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.903335 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.903058    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.903821 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.903795    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.904457 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.904434    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e716825-de92-4c9d-9b36-007ea4218a98-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.907838 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.905628    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.919097 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.919067    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e716825-de92-4c9d-9b36-007ea4218a98-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.941004 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.940975    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e716825-de92-4c9d-9b36-007ea4218a98-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:20.941117 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:20.941079    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9whx\" (UniqueName: \"kubernetes.io/projected/4e716825-de92-4c9d-9b36-007ea4218a98-kube-api-access-t9whx\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:21.011124 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:21.011094    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:21.284336 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:21.284301    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"]
Oct 30 04:17:21.284824 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:21.284517    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb" podUID="425e4e27-91f5-47b0-b8cf-311b5550e48a" containerName="istio-proxy" containerID="cri-o://739fab602a02e6c6af1d1e23b5a9826a98f02b427e58769a710c99ca5ba7dfae" gracePeriod=30
Oct 30 04:17:21.330490 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:21.330457    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"]
Oct 30 04:17:21.333890 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:17:21.333857    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e716825_de92_4c9d_9b36_007ea4218a98.slice/crio-6067be3afe85426ba4618bd66f8cb35bcc5e75c7a8b6a2adc391a5fd7a3d89c7 WatchSource:0}: Error finding container 6067be3afe85426ba4618bd66f8cb35bcc5e75c7a8b6a2adc391a5fd7a3d89c7: Status 404 returned error can't find the container with id 6067be3afe85426ba4618bd66f8cb35bcc5e75c7a8b6a2adc391a5fd7a3d89c7
Oct 30 04:17:21.336219 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:21.336190    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:21.336285 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:21.336271    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:21.336329 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:21.336301    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:21.418427 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:21.418319    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj" event={"ID":"10b28e83-cdc5-4591-858b-80df98885d4e","Type":"ContainerStarted","Data":"a3938d264c0c2e896b85ff65153cf0422811442940c3ab85b93c372e65b1cd62"}
Oct 30 04:17:21.418427 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:21.418386    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj" event={"ID":"10b28e83-cdc5-4591-858b-80df98885d4e","Type":"ContainerStarted","Data":"ad73c7d3a82c333a7fec58bfed95f090878db2f692b8514459abc2e9d650b57a"}
Oct 30 04:17:21.426398 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:21.426359    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s" event={"ID":"4e716825-de92-4c9d-9b36-007ea4218a98","Type":"ContainerStarted","Data":"6067be3afe85426ba4618bd66f8cb35bcc5e75c7a8b6a2adc391a5fd7a3d89c7"}
Oct 30 04:17:21.463568 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:21.463505    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj" podStartSLOduration=1.4634854480000001 podStartE2EDuration="1.463485448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:21.462131856 +0000 UTC m=+1160.874527429" watchObservedRunningTime="2026-10-30 04:17:21.463485448 +0000 UTC m=+1160.875881024"
Oct 30 04:17:21.566280 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:21.566185    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:21.585558 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:21.585102    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:22.339141 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:22.339103    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:17:22.433639 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:22.433595    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s" event={"ID":"4e716825-de92-4c9d-9b36-007ea4218a98","Type":"ContainerStarted","Data":"2f09cbdac2cbd04e842944626106cc89e120926fdef07ae3a2074d673c42284d"}
Oct 30 04:17:22.433886 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:22.433860    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:22.434798 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:22.434778    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:22.478940 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:22.478894    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s" podStartSLOduration=2.478880384 podStartE2EDuration="2.478880384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:22.477528058 +0000 UTC m=+1161.889923631" watchObservedRunningTime="2026-10-30 04:17:22.478880384 +0000 UTC m=+1161.891275957"
Oct 30 04:17:23.011316 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:23.011282    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:23.016706 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:23.016675    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:23.331480 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:23.331386    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56"]
Oct 30 04:17:23.331624 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:23.331587    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56" podUID="416f0a2e-bd70-499c-bafe-0b85913694c5" containerName="maas-api" containerID="cri-o://4318fa7f9d22b6a73f5be01441abf7255715969f4debc3e9e5b3ea9c1ca83166" gracePeriod=30
Oct 30 04:17:23.439975 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:23.439945    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:23.440896 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:23.440874    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:23.584736 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:23.584659    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56"
Oct 30 04:17:23.629224 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:23.629193    4874 patch_prober.go:29] interesting pod/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 30 04:17:23.629402 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:23.629258    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7" podUID="35fb8fc8-50a2-409b-9c88-8ad5219ace64" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:17:23.655686 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:23.655655    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/416f0a2e-bd70-499c-bafe-0b85913694c5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416f0a2e-bd70-499c-bafe-0b85913694c5-openshift-service-ca\") pod \"416f0a2e-bd70-499c-bafe-0b85913694c5\" (UID: \"416f0a2e-bd70-499c-bafe-0b85913694c5\") "
Oct 30 04:17:23.655854 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:23.655719    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/416f0a2e-bd70-499c-bafe-0b85913694c5-kube-api-access-jwvkq\" (UniqueName: \"kubernetes.io/projected/416f0a2e-bd70-499c-bafe-0b85913694c5-kube-api-access-jwvkq\") pod \"416f0a2e-bd70-499c-bafe-0b85913694c5\" (UID: \"416f0a2e-bd70-499c-bafe-0b85913694c5\") "
Oct 30 04:17:23.655854 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:23.655809    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/416f0a2e-bd70-499c-bafe-0b85913694c5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416f0a2e-bd70-499c-bafe-0b85913694c5-maas-api-tls\") pod \"416f0a2e-bd70-499c-bafe-0b85913694c5\" (UID: \"416f0a2e-bd70-499c-bafe-0b85913694c5\") "
Oct 30 04:17:23.655975 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:23.655889    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/416f0a2e-bd70-499c-bafe-0b85913694c5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/416f0a2e-bd70-499c-bafe-0b85913694c5-metrics-certs\") pod \"416f0a2e-bd70-499c-bafe-0b85913694c5\" (UID: \"416f0a2e-bd70-499c-bafe-0b85913694c5\") "
Oct 30 04:17:23.656112 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:23.656082    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416f0a2e-bd70-499c-bafe-0b85913694c5-openshift-service-ca" pod "416f0a2e-bd70-499c-bafe-0b85913694c5" (UID: "416f0a2e-bd70-499c-bafe-0b85913694c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:23.656323 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:23.656287    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/416f0a2e-bd70-499c-bafe-0b85913694c5-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:23.658623 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:23.658596    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416f0a2e-bd70-499c-bafe-0b85913694c5-kube-api-access-jwvkq" pod "416f0a2e-bd70-499c-bafe-0b85913694c5" (UID: "416f0a2e-bd70-499c-bafe-0b85913694c5"). InnerVolumeSpecName "kube-api-access-jwvkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:23.660548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:23.660522    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/416f0a2e-bd70-499c-bafe-0b85913694c5-maas-api-tls" pod "416f0a2e-bd70-499c-bafe-0b85913694c5" (UID: "416f0a2e-bd70-499c-bafe-0b85913694c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:23.660647 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:23.660603    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/416f0a2e-bd70-499c-bafe-0b85913694c5-metrics-certs" pod "416f0a2e-bd70-499c-bafe-0b85913694c5" (UID: "416f0a2e-bd70-499c-bafe-0b85913694c5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:23.757142 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:23.757113    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwvkq\" (UniqueName: \"kubernetes.io/projected/416f0a2e-bd70-499c-bafe-0b85913694c5-kube-api-access-jwvkq\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:23.757142 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:23.757137    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/416f0a2e-bd70-499c-bafe-0b85913694c5-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:23.757142 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:23.757147    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/416f0a2e-bd70-499c-bafe-0b85913694c5-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:24.164806 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.164782    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:17:24.299548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.277564    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35fb8fc8-50a2-409b-9c88-8ad5219ace64-kube-api-access-f8x46\" (UniqueName: \"kubernetes.io/projected/35fb8fc8-50a2-409b-9c88-8ad5219ace64-kube-api-access-f8x46\") pod \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") "
Oct 30 04:17:24.299548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.293698    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35fb8fc8-50a2-409b-9c88-8ad5219ace64-kube-api-access-f8x46" pod "35fb8fc8-50a2-409b-9c88-8ad5219ace64" (UID: "35fb8fc8-50a2-409b-9c88-8ad5219ace64"). InnerVolumeSpecName "kube-api-access-f8x46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:24.299548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.294370    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-workload-socket\") pod \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") "
Oct 30 04:17:24.299548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.294456    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-credential-socket\") pod \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") "
Oct 30 04:17:24.299548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.294507    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-envoy\") pod \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") "
Oct 30 04:17:24.299548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.294629    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-workload-socket" pod "35fb8fc8-50a2-409b-9c88-8ad5219ace64" (UID: "35fb8fc8-50a2-409b-9c88-8ad5219ace64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:24.299548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.294815    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-credential-socket" pod "35fb8fc8-50a2-409b-9c88-8ad5219ace64" (UID: "35fb8fc8-50a2-409b-9c88-8ad5219ace64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:24.299548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.294818    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-data\") pod \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") "
Oct 30 04:17:24.299548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.294977    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-token\" (UniqueName: \"kubernetes.io/projected/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-token\") pod \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") "
Oct 30 04:17:24.299548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.294998    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-data" pod "35fb8fc8-50a2-409b-9c88-8ad5219ace64" (UID: "35fb8fc8-50a2-409b-9c88-8ad5219ace64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:24.299548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.295009    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-podinfo\") pod \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") "
Oct 30 04:17:24.299548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.295066    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istiod-ca-cert\") pod \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") "
Oct 30 04:17:24.299548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.295118    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-workload-certs\") pod \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\" (UID: \"35fb8fc8-50a2-409b-9c88-8ad5219ace64\") "
Oct 30 04:17:24.299548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.295528    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8x46\" (UniqueName: \"kubernetes.io/projected/35fb8fc8-50a2-409b-9c88-8ad5219ace64-kube-api-access-f8x46\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:24.299548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.295549    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:24.299548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.295564    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:24.299548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.295577    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:24.299548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.295838    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-workload-certs" pod "35fb8fc8-50a2-409b-9c88-8ad5219ace64" (UID: "35fb8fc8-50a2-409b-9c88-8ad5219ace64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:24.299548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.295846    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istiod-ca-cert" pod "35fb8fc8-50a2-409b-9c88-8ad5219ace64" (UID: "35fb8fc8-50a2-409b-9c88-8ad5219ace64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:24.299548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.297509    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-token" pod "35fb8fc8-50a2-409b-9c88-8ad5219ace64" (UID: "35fb8fc8-50a2-409b-9c88-8ad5219ace64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:24.299548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.297639    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-envoy" pod "35fb8fc8-50a2-409b-9c88-8ad5219ace64" (UID: "35fb8fc8-50a2-409b-9c88-8ad5219ace64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:24.299548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.298864    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-podinfo" pod "35fb8fc8-50a2-409b-9c88-8ad5219ace64" (UID: "35fb8fc8-50a2-409b-9c88-8ad5219ace64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:17:24.396582 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.396557    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:24.396582 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.396583    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:24.396702 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.396591    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:24.396702 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.396599    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:24.396702 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.396607    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35fb8fc8-50a2-409b-9c88-8ad5219ace64-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:24.446351 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.446317    4874 generic.go:356] "Generic (PLEG): container finished" podID="416f0a2e-bd70-499c-bafe-0b85913694c5" containerID="4318fa7f9d22b6a73f5be01441abf7255715969f4debc3e9e5b3ea9c1ca83166" exitCode=0
Oct 30 04:17:24.446723 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.446372    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56"
Oct 30 04:17:24.446723 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.446399    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56" event={"ID":"416f0a2e-bd70-499c-bafe-0b85913694c5","Type":"ContainerDied","Data":"4318fa7f9d22b6a73f5be01441abf7255715969f4debc3e9e5b3ea9c1ca83166"}
Oct 30 04:17:24.446723 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.446450    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56" event={"ID":"416f0a2e-bd70-499c-bafe-0b85913694c5","Type":"ContainerDied","Data":"479ae8b1a846fc37ec68eecf84c6eb2e90bc2517f0c5b2f46eca05c381c51de8"}
Oct 30 04:17:24.446723 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.446472    4874 scope.go:122] "RemoveContainer" containerID="4318fa7f9d22b6a73f5be01441abf7255715969f4debc3e9e5b3ea9c1ca83166"
Oct 30 04:17:24.448342 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.448318    4874 generic.go:356] "Generic (PLEG): container finished" podID="35fb8fc8-50a2-409b-9c88-8ad5219ace64" containerID="d89c4371bf215aa07459c67a18c84b85062538d641f06aa80568a5b8f6108c70" exitCode=0
Oct 30 04:17:24.448435 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.448356    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"
Oct 30 04:17:24.448435 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.448393    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7" event={"ID":"35fb8fc8-50a2-409b-9c88-8ad5219ace64","Type":"ContainerDied","Data":"d89c4371bf215aa07459c67a18c84b85062538d641f06aa80568a5b8f6108c70"}
Oct 30 04:17:24.448435 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.448432    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7" event={"ID":"35fb8fc8-50a2-409b-9c88-8ad5219ace64","Type":"ContainerDied","Data":"e73e24a6c67f025e3b3bbc62341336728c33206b8bc3538589cdb9a3c90107c8"}
Oct 30 04:17:24.484290 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.484269    4874 scope.go:122] "RemoveContainer" containerID="4318fa7f9d22b6a73f5be01441abf7255715969f4debc3e9e5b3ea9c1ca83166"
Oct 30 04:17:24.484584 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:17:24.484565    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4318fa7f9d22b6a73f5be01441abf7255715969f4debc3e9e5b3ea9c1ca83166\": container with ID starting with 4318fa7f9d22b6a73f5be01441abf7255715969f4debc3e9e5b3ea9c1ca83166 not found: ID does not exist" containerID="4318fa7f9d22b6a73f5be01441abf7255715969f4debc3e9e5b3ea9c1ca83166"
Oct 30 04:17:24.484643 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.484591    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4318fa7f9d22b6a73f5be01441abf7255715969f4debc3e9e5b3ea9c1ca83166"} err="failed to get container status \"4318fa7f9d22b6a73f5be01441abf7255715969f4debc3e9e5b3ea9c1ca83166\": rpc error: code = NotFound desc = could not find container \"4318fa7f9d22b6a73f5be01441abf7255715969f4debc3e9e5b3ea9c1ca83166\": container with ID starting with 4318fa7f9d22b6a73f5be01441abf7255715969f4debc3e9e5b3ea9c1ca83166 not found: ID does not exist"
Oct 30 04:17:24.484643 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.484607    4874 scope.go:122] "RemoveContainer" containerID="d89c4371bf215aa07459c67a18c84b85062538d641f06aa80568a5b8f6108c70"
Oct 30 04:17:24.492337 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.492316    4874 scope.go:122] "RemoveContainer" containerID="c3f7ec968483d8d7f1cb6bfd6cd13a41e82fb06d023e9779ab508858454edb20"
Oct 30 04:17:24.498313 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.498291    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56"]
Oct 30 04:17:24.500259 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.500225    4874 scope.go:122] "RemoveContainer" containerID="d89c4371bf215aa07459c67a18c84b85062538d641f06aa80568a5b8f6108c70"
Oct 30 04:17:24.500504 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:17:24.500487    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d89c4371bf215aa07459c67a18c84b85062538d641f06aa80568a5b8f6108c70\": container with ID starting with d89c4371bf215aa07459c67a18c84b85062538d641f06aa80568a5b8f6108c70 not found: ID does not exist" containerID="d89c4371bf215aa07459c67a18c84b85062538d641f06aa80568a5b8f6108c70"
Oct 30 04:17:24.500561 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.500514    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89c4371bf215aa07459c67a18c84b85062538d641f06aa80568a5b8f6108c70"} err="failed to get container status \"d89c4371bf215aa07459c67a18c84b85062538d641f06aa80568a5b8f6108c70\": rpc error: code = NotFound desc = could not find container \"d89c4371bf215aa07459c67a18c84b85062538d641f06aa80568a5b8f6108c70\": container with ID starting with d89c4371bf215aa07459c67a18c84b85062538d641f06aa80568a5b8f6108c70 not found: ID does not exist"
Oct 30 04:17:24.500561 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.500531    4874 scope.go:122] "RemoveContainer" containerID="c3f7ec968483d8d7f1cb6bfd6cd13a41e82fb06d023e9779ab508858454edb20"
Oct 30 04:17:24.500750 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:17:24.500735    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3f7ec968483d8d7f1cb6bfd6cd13a41e82fb06d023e9779ab508858454edb20\": container with ID starting with c3f7ec968483d8d7f1cb6bfd6cd13a41e82fb06d023e9779ab508858454edb20 not found: ID does not exist" containerID="c3f7ec968483d8d7f1cb6bfd6cd13a41e82fb06d023e9779ab508858454edb20"
Oct 30 04:17:24.500800 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.500757    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3f7ec968483d8d7f1cb6bfd6cd13a41e82fb06d023e9779ab508858454edb20"} err="failed to get container status \"c3f7ec968483d8d7f1cb6bfd6cd13a41e82fb06d023e9779ab508858454edb20\": rpc error: code = NotFound desc = could not find container \"c3f7ec968483d8d7f1cb6bfd6cd13a41e82fb06d023e9779ab508858454edb20\": container with ID starting with c3f7ec968483d8d7f1cb6bfd6cd13a41e82fb06d023e9779ab508858454edb20 not found: ID does not exist"
Oct 30 04:17:24.506942 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.506918    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ad07bf73-75d44cc7fc-z5d56"]
Oct 30 04:17:24.547912 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.547876    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"]
Oct 30 04:17:24.552775 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.552713    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-eb861bd9-openshift-default-7846766d9f-99hd7"]
Oct 30 04:17:24.622317 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.622294    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:17:24.699779 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.699743    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a09d87c-a553-4a74-8559-59141d1778a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a09d87c-a553-4a74-8559-59141d1778a4-istiod-ca-cert\") pod \"1a09d87c-a553-4a74-8559-59141d1778a4\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") "
Oct 30 04:17:24.699914 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.699804    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a09d87c-a553-4a74-8559-59141d1778a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a09d87c-a553-4a74-8559-59141d1778a4-istio-podinfo\") pod \"1a09d87c-a553-4a74-8559-59141d1778a4\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") "
Oct 30 04:17:24.699914 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.699854    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a09d87c-a553-4a74-8559-59141d1778a4-istio-token\" (UniqueName: \"kubernetes.io/projected/1a09d87c-a553-4a74-8559-59141d1778a4-istio-token\") pod \"1a09d87c-a553-4a74-8559-59141d1778a4\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") "
Oct 30 04:17:24.699914 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.699884    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-istio-envoy\") pod \"1a09d87c-a553-4a74-8559-59141d1778a4\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") "
Oct 30 04:17:24.700107 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.699949    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-workload-certs\") pod \"1a09d87c-a553-4a74-8559-59141d1778a4\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") "
Oct 30 04:17:24.700107 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.699977    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-istio-data\") pod \"1a09d87c-a553-4a74-8559-59141d1778a4\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") "
Oct 30 04:17:24.700107 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.700043    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a09d87c-a553-4a74-8559-59141d1778a4-kube-api-access-bdm8r\" (UniqueName: \"kubernetes.io/projected/1a09d87c-a553-4a74-8559-59141d1778a4-kube-api-access-bdm8r\") pod \"1a09d87c-a553-4a74-8559-59141d1778a4\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") "
Oct 30 04:17:24.700107 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.700082    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-credential-socket\") pod \"1a09d87c-a553-4a74-8559-59141d1778a4\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") "
Oct 30 04:17:24.700341 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.700118    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-workload-socket\") pod \"1a09d87c-a553-4a74-8559-59141d1778a4\" (UID: \"1a09d87c-a553-4a74-8559-59141d1778a4\") "
Oct 30 04:17:24.700341 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.700143    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a09d87c-a553-4a74-8559-59141d1778a4-istiod-ca-cert" pod "1a09d87c-a553-4a74-8559-59141d1778a4" (UID: "1a09d87c-a553-4a74-8559-59141d1778a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:24.700341 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.700254    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-istio-data" pod "1a09d87c-a553-4a74-8559-59141d1778a4" (UID: "1a09d87c-a553-4a74-8559-59141d1778a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:24.700611 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.700460    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-workload-socket" pod "1a09d87c-a553-4a74-8559-59141d1778a4" (UID: "1a09d87c-a553-4a74-8559-59141d1778a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:24.700611 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.700474    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-credential-socket" pod "1a09d87c-a553-4a74-8559-59141d1778a4" (UID: "1a09d87c-a553-4a74-8559-59141d1778a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:24.700611 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.700503    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:24.700611 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.700525    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a09d87c-a553-4a74-8559-59141d1778a4-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:24.700816 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.700650    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-workload-certs" pod "1a09d87c-a553-4a74-8559-59141d1778a4" (UID: "1a09d87c-a553-4a74-8559-59141d1778a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:24.702582 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.702555    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-istio-envoy" pod "1a09d87c-a553-4a74-8559-59141d1778a4" (UID: "1a09d87c-a553-4a74-8559-59141d1778a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:24.702689 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.702615    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a09d87c-a553-4a74-8559-59141d1778a4-istio-podinfo" pod "1a09d87c-a553-4a74-8559-59141d1778a4" (UID: "1a09d87c-a553-4a74-8559-59141d1778a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:17:24.702689 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.702620    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a09d87c-a553-4a74-8559-59141d1778a4-istio-token" pod "1a09d87c-a553-4a74-8559-59141d1778a4" (UID: "1a09d87c-a553-4a74-8559-59141d1778a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:24.702689 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.702619    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a09d87c-a553-4a74-8559-59141d1778a4-kube-api-access-bdm8r" pod "1a09d87c-a553-4a74-8559-59141d1778a4" (UID: "1a09d87c-a553-4a74-8559-59141d1778a4"). InnerVolumeSpecName "kube-api-access-bdm8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:24.726870 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.726844    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35fb8fc8-50a2-409b-9c88-8ad5219ace64" path="/var/lib/kubelet/pods/35fb8fc8-50a2-409b-9c88-8ad5219ace64/volumes"
Oct 30 04:17:24.727232 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.727219    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="416f0a2e-bd70-499c-bafe-0b85913694c5" path="/var/lib/kubelet/pods/416f0a2e-bd70-499c-bafe-0b85913694c5/volumes"
Oct 30 04:17:24.801064 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.801039    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a09d87c-a553-4a74-8559-59141d1778a4-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:24.801064 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.801063    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a09d87c-a553-4a74-8559-59141d1778a4-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:24.801174 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.801072    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:24.801174 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.801080    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:24.801174 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.801088    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdm8r\" (UniqueName: \"kubernetes.io/projected/1a09d87c-a553-4a74-8559-59141d1778a4-kube-api-access-bdm8r\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:24.801174 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.801097    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:24.801174 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:24.801105    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a09d87c-a553-4a74-8559-59141d1778a4-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:25.456938 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:25.456905    4874 generic.go:356] "Generic (PLEG): container finished" podID="1a09d87c-a553-4a74-8559-59141d1778a4" containerID="4580303c965b08c3cf86113f4b99df03b1dd9503da30078554a994abc5a88958" exitCode=0
Oct 30 04:17:25.457353 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:25.456959    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"
Oct 30 04:17:25.457353 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:25.456986    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d" event={"ID":"1a09d87c-a553-4a74-8559-59141d1778a4","Type":"ContainerDied","Data":"4580303c965b08c3cf86113f4b99df03b1dd9503da30078554a994abc5a88958"}
Oct 30 04:17:25.457353 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:25.457034    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d" event={"ID":"1a09d87c-a553-4a74-8559-59141d1778a4","Type":"ContainerDied","Data":"5b59bf29dcc77850a270efb6ff4f6735389ce1d267f84df6b0efc84668a267ca"}
Oct 30 04:17:25.457353 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:25.457055    4874 scope.go:122] "RemoveContainer" containerID="4580303c965b08c3cf86113f4b99df03b1dd9503da30078554a994abc5a88958"
Oct 30 04:17:25.465481 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:25.465463    4874 scope.go:122] "RemoveContainer" containerID="143a21015aba4d23c136bdb21e96d6e6ae261d35a94e4673bb0ee466137aa36c"
Oct 30 04:17:25.473216 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:25.473198    4874 scope.go:122] "RemoveContainer" containerID="4580303c965b08c3cf86113f4b99df03b1dd9503da30078554a994abc5a88958"
Oct 30 04:17:25.473513 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:17:25.473490    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4580303c965b08c3cf86113f4b99df03b1dd9503da30078554a994abc5a88958\": container with ID starting with 4580303c965b08c3cf86113f4b99df03b1dd9503da30078554a994abc5a88958 not found: ID does not exist" containerID="4580303c965b08c3cf86113f4b99df03b1dd9503da30078554a994abc5a88958"
Oct 30 04:17:25.473560 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:25.473521    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4580303c965b08c3cf86113f4b99df03b1dd9503da30078554a994abc5a88958"} err="failed to get container status \"4580303c965b08c3cf86113f4b99df03b1dd9503da30078554a994abc5a88958\": rpc error: code = NotFound desc = could not find container \"4580303c965b08c3cf86113f4b99df03b1dd9503da30078554a994abc5a88958\": container with ID starting with 4580303c965b08c3cf86113f4b99df03b1dd9503da30078554a994abc5a88958 not found: ID does not exist"
Oct 30 04:17:25.473560 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:25.473538    4874 scope.go:122] "RemoveContainer" containerID="143a21015aba4d23c136bdb21e96d6e6ae261d35a94e4673bb0ee466137aa36c"
Oct 30 04:17:25.473778 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:17:25.473758    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"143a21015aba4d23c136bdb21e96d6e6ae261d35a94e4673bb0ee466137aa36c\": container with ID starting with 143a21015aba4d23c136bdb21e96d6e6ae261d35a94e4673bb0ee466137aa36c not found: ID does not exist" containerID="143a21015aba4d23c136bdb21e96d6e6ae261d35a94e4673bb0ee466137aa36c"
Oct 30 04:17:25.473817 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:25.473783    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"143a21015aba4d23c136bdb21e96d6e6ae261d35a94e4673bb0ee466137aa36c"} err="failed to get container status \"143a21015aba4d23c136bdb21e96d6e6ae261d35a94e4673bb0ee466137aa36c\": rpc error: code = NotFound desc = could not find container \"143a21015aba4d23c136bdb21e96d6e6ae261d35a94e4673bb0ee466137aa36c\": container with ID starting with 143a21015aba4d23c136bdb21e96d6e6ae261d35a94e4673bb0ee466137aa36c not found: ID does not exist"
Oct 30 04:17:25.516834 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:25.516806    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"]
Oct 30 04:17:25.528230 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:25.528210    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-566c3724-gw-openshift-default-777b96bbdf-zh77d"]
Oct 30 04:17:25.608871 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:17:25.608838    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:17:26.300396 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.300361    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7"]
Oct 30 04:17:26.300778 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.300761    4874 cpu_manager.go:414] "containerMap: removing container" podUID="35fb8fc8-50a2-409b-9c88-8ad5219ace64" containerName="istio-proxy"
Oct 30 04:17:26.300872 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.300780    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35fb8fc8-50a2-409b-9c88-8ad5219ace64" containerName="istio-proxy"
Oct 30 04:17:26.300872 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.300798    4874 cpu_manager.go:414] "containerMap: removing container" podUID="1a09d87c-a553-4a74-8559-59141d1778a4" containerName="istio-proxy"
Oct 30 04:17:26.300872 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.300807    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a09d87c-a553-4a74-8559-59141d1778a4" containerName="istio-proxy"
Oct 30 04:17:26.300872 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.300830    4874 cpu_manager.go:414] "containerMap: removing container" podUID="1a09d87c-a553-4a74-8559-59141d1778a4" containerName="istio-proxy"
Oct 30 04:17:26.300872 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.300838    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a09d87c-a553-4a74-8559-59141d1778a4" containerName="istio-proxy"
Oct 30 04:17:26.300872 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.300849    4874 cpu_manager.go:414] "containerMap: removing container" podUID="416f0a2e-bd70-499c-bafe-0b85913694c5" containerName="maas-api"
Oct 30 04:17:26.300872 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.300857    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="416f0a2e-bd70-499c-bafe-0b85913694c5" containerName="maas-api"
Oct 30 04:17:26.301227 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.300954    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="35fb8fc8-50a2-409b-9c88-8ad5219ace64" containerName="istio-proxy"
Oct 30 04:17:26.301227 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.300968    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a09d87c-a553-4a74-8559-59141d1778a4" containerName="istio-proxy"
Oct 30 04:17:26.301227 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.300980    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a09d87c-a553-4a74-8559-59141d1778a4" containerName="istio-proxy"
Oct 30 04:17:26.301227 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.300989    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="416f0a2e-bd70-499c-bafe-0b85913694c5" containerName="maas-api"
Oct 30 04:17:26.301227 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.300999    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="35fb8fc8-50a2-409b-9c88-8ad5219ace64" containerName="istio-proxy"
Oct 30 04:17:26.303339 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.303309    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7"
Oct 30 04:17:26.319175 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.319157    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-5d74b9-dockercfg-rkrs2\""
Oct 30 04:17:26.322355 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.322335    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-5d74b9\""
Oct 30 04:17:26.324382 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.324361    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7"]
Oct 30 04:17:26.400068 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.400031    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh"]
Oct 30 04:17:26.405426 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.405393    4874 cpu_manager.go:414] "containerMap: removing container" podUID="35fb8fc8-50a2-409b-9c88-8ad5219ace64" containerName="istio-proxy"
Oct 30 04:17:26.405632 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.405613    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35fb8fc8-50a2-409b-9c88-8ad5219ace64" containerName="istio-proxy"
Oct 30 04:17:26.413030 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.412997    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh"
Oct 30 04:17:26.416838 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.416803    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b386201-0f5e-4add-98a8-bb3bcf476089-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7\" (UID: \"5b386201-0f5e-4add-98a8-bb3bcf476089\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7"
Oct 30 04:17:26.417008 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.416874    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4lgz\" (UniqueName: \"kubernetes.io/projected/5b386201-0f5e-4add-98a8-bb3bcf476089-kube-api-access-c4lgz\") pod \"payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7\" (UID: \"5b386201-0f5e-4add-98a8-bb3bcf476089\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7"
Oct 30 04:17:26.425073 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.425001    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh"]
Oct 30 04:17:26.464297 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.463985    4874 generic.go:356] "Generic (PLEG): container finished" podID="425e4e27-91f5-47b0-b8cf-311b5550e48a" containerID="739fab602a02e6c6af1d1e23b5a9826a98f02b427e58769a710c99ca5ba7dfae" exitCode=0
Oct 30 04:17:26.464297 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.464116    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb" event={"ID":"425e4e27-91f5-47b0-b8cf-311b5550e48a","Type":"ContainerDied","Data":"739fab602a02e6c6af1d1e23b5a9826a98f02b427e58769a710c99ca5ba7dfae"}
Oct 30 04:17:26.464297 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.464165    4874 scope.go:122] "RemoveContainer" containerID="d34846516b9345867589e5fdcf7f8e186eed965b05d45540178502c919f6f11f"
Oct 30 04:17:26.518005 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.517973    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b386201-0f5e-4add-98a8-bb3bcf476089-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7\" (UID: \"5b386201-0f5e-4add-98a8-bb3bcf476089\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7"
Oct 30 04:17:26.518129 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.518034    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1774f86-5e25-4358-b250-d26c6ea6c601-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh\" (UID: \"e1774f86-5e25-4358-b250-d26c6ea6c601\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh"
Oct 30 04:17:26.518129 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.518083    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4lgz\" (UniqueName: \"kubernetes.io/projected/5b386201-0f5e-4add-98a8-bb3bcf476089-kube-api-access-c4lgz\") pod \"payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7\" (UID: \"5b386201-0f5e-4add-98a8-bb3bcf476089\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7"
Oct 30 04:17:26.518298 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.518273    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vbh4\" (UniqueName: \"kubernetes.io/projected/e1774f86-5e25-4358-b250-d26c6ea6c601-kube-api-access-7vbh4\") pod \"payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh\" (UID: \"e1774f86-5e25-4358-b250-d26c6ea6c601\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh"
Oct 30 04:17:26.518710 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.518693    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b386201-0f5e-4add-98a8-bb3bcf476089-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7\" (UID: \"5b386201-0f5e-4add-98a8-bb3bcf476089\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7"
Oct 30 04:17:26.540303 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.540284    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4lgz\" (UniqueName: \"kubernetes.io/projected/5b386201-0f5e-4add-98a8-bb3bcf476089-kube-api-access-c4lgz\") pod \"payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7\" (UID: \"5b386201-0f5e-4add-98a8-bb3bcf476089\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7"
Oct 30 04:17:26.557587 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.557537    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:17:26.619106 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.619082    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vbh4\" (UniqueName: \"kubernetes.io/projected/e1774f86-5e25-4358-b250-d26c6ea6c601-kube-api-access-7vbh4\") pod \"payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh\" (UID: \"e1774f86-5e25-4358-b250-d26c6ea6c601\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh"
Oct 30 04:17:26.619218 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.619156    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1774f86-5e25-4358-b250-d26c6ea6c601-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh\" (UID: \"e1774f86-5e25-4358-b250-d26c6ea6c601\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh"
Oct 30 04:17:26.619743 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.619721    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1774f86-5e25-4358-b250-d26c6ea6c601-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh\" (UID: \"e1774f86-5e25-4358-b250-d26c6ea6c601\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh"
Oct 30 04:17:26.641350 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.641329    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vbh4\" (UniqueName: \"kubernetes.io/projected/e1774f86-5e25-4358-b250-d26c6ea6c601-kube-api-access-7vbh4\") pod \"payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh\" (UID: \"e1774f86-5e25-4358-b250-d26c6ea6c601\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh"
Oct 30 04:17:26.682593 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.682566    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7"
Oct 30 04:17:26.720450 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.720426    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/425e4e27-91f5-47b0-b8cf-311b5550e48a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/425e4e27-91f5-47b0-b8cf-311b5550e48a-istiod-ca-cert\") pod \"425e4e27-91f5-47b0-b8cf-311b5550e48a\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") "
Oct 30 04:17:26.720565 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.720459    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-podinfo\") pod \"425e4e27-91f5-47b0-b8cf-311b5550e48a\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") "
Oct 30 04:17:26.720669 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.720648    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-data\") pod \"425e4e27-91f5-47b0-b8cf-311b5550e48a\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") "
Oct 30 04:17:26.720751 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.720734    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/425e4e27-91f5-47b0-b8cf-311b5550e48a-kube-api-access-g7l2g\" (UniqueName: \"kubernetes.io/projected/425e4e27-91f5-47b0-b8cf-311b5550e48a-kube-api-access-g7l2g\") pod \"425e4e27-91f5-47b0-b8cf-311b5550e48a\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") "
Oct 30 04:17:26.720793 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.720770    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/425e4e27-91f5-47b0-b8cf-311b5550e48a-istiod-ca-cert" pod "425e4e27-91f5-47b0-b8cf-311b5550e48a" (UID: "425e4e27-91f5-47b0-b8cf-311b5550e48a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:26.720831 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.720784    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-workload-certs\") pod \"425e4e27-91f5-47b0-b8cf-311b5550e48a\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") "
Oct 30 04:17:26.720893 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.720876    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-data" pod "425e4e27-91f5-47b0-b8cf-311b5550e48a" (UID: "425e4e27-91f5-47b0-b8cf-311b5550e48a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:26.720955 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.720942    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-envoy\") pod \"425e4e27-91f5-47b0-b8cf-311b5550e48a\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") "
Oct 30 04:17:26.721021 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.720970    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-token\" (UniqueName: \"kubernetes.io/projected/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-token\") pod \"425e4e27-91f5-47b0-b8cf-311b5550e48a\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") "
Oct 30 04:17:26.721021 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.720970    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-workload-certs" pod "425e4e27-91f5-47b0-b8cf-311b5550e48a" (UID: "425e4e27-91f5-47b0-b8cf-311b5550e48a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:26.721125 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.721024    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-workload-socket\") pod \"425e4e27-91f5-47b0-b8cf-311b5550e48a\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") "
Oct 30 04:17:26.721125 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.721085    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-credential-socket\") pod \"425e4e27-91f5-47b0-b8cf-311b5550e48a\" (UID: \"425e4e27-91f5-47b0-b8cf-311b5550e48a\") "
Oct 30 04:17:26.721375 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.721330    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-workload-socket" pod "425e4e27-91f5-47b0-b8cf-311b5550e48a" (UID: "425e4e27-91f5-47b0-b8cf-311b5550e48a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:26.721464 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.721384    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-credential-socket" pod "425e4e27-91f5-47b0-b8cf-311b5550e48a" (UID: "425e4e27-91f5-47b0-b8cf-311b5550e48a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:26.721535 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.721494    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:26.721535 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.721512    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:26.721535 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.721528    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/425e4e27-91f5-47b0-b8cf-311b5550e48a-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:26.721699 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.721541    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:26.721699 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.721554    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:26.724810 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.723087    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-envoy" pod "425e4e27-91f5-47b0-b8cf-311b5550e48a" (UID: "425e4e27-91f5-47b0-b8cf-311b5550e48a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:26.724810 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.723258    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-token" pod "425e4e27-91f5-47b0-b8cf-311b5550e48a" (UID: "425e4e27-91f5-47b0-b8cf-311b5550e48a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:26.724810 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.723327    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/425e4e27-91f5-47b0-b8cf-311b5550e48a-kube-api-access-g7l2g" pod "425e4e27-91f5-47b0-b8cf-311b5550e48a" (UID: "425e4e27-91f5-47b0-b8cf-311b5550e48a"). InnerVolumeSpecName "kube-api-access-g7l2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:26.724810 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.723361    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-podinfo" pod "425e4e27-91f5-47b0-b8cf-311b5550e48a" (UID: "425e4e27-91f5-47b0-b8cf-311b5550e48a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:17:26.730545 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.730521    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh"
Oct 30 04:17:26.740881 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.740851    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a09d87c-a553-4a74-8559-59141d1778a4" path="/var/lib/kubelet/pods/1a09d87c-a553-4a74-8559-59141d1778a4/volumes"
Oct 30 04:17:26.822905 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.822844    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:26.822905 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.822870    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:26.822905 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.822880    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/425e4e27-91f5-47b0-b8cf-311b5550e48a-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:26.822905 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.822889    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7l2g\" (UniqueName: \"kubernetes.io/projected/425e4e27-91f5-47b0-b8cf-311b5550e48a-kube-api-access-g7l2g\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:26.984644 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:26.984553    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7"]
Oct 30 04:17:26.987121 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:17:26.987093    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b386201_0f5e_4add_98a8_bb3bcf476089.slice/crio-0d6a8888e43b048b4fa2ff7068c549bc39ff94b1185c610485e85f811b369478 WatchSource:0}: Error finding container 0d6a8888e43b048b4fa2ff7068c549bc39ff94b1185c610485e85f811b369478: Status 404 returned error can't find the container with id 0d6a8888e43b048b4fa2ff7068c549bc39ff94b1185c610485e85f811b369478
Oct 30 04:17:27.042821 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.042794    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:17:27.054225 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.054197    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh"]
Oct 30 04:17:27.057549 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:17:27.057524    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1774f86_5e25_4358_b250_d26c6ea6c601.slice/crio-b4212c87e6403b78797fc3bb48bc0f90ec2c30e1467a6fae587c32943bc6ba14 WatchSource:0}: Error finding container b4212c87e6403b78797fc3bb48bc0f90ec2c30e1467a6fae587c32943bc6ba14: Status 404 returned error can't find the container with id b4212c87e6403b78797fc3bb48bc0f90ec2c30e1467a6fae587c32943bc6ba14
Oct 30 04:17:27.160460 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.160382    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"]
Oct 30 04:17:27.160893 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.160874    4874 cpu_manager.go:414] "containerMap: removing container" podUID="425e4e27-91f5-47b0-b8cf-311b5550e48a" containerName="istio-proxy"
Oct 30 04:17:27.161009 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.160895    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="425e4e27-91f5-47b0-b8cf-311b5550e48a" containerName="istio-proxy"
Oct 30 04:17:27.161009 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.160915    4874 cpu_manager.go:414] "containerMap: removing container" podUID="425e4e27-91f5-47b0-b8cf-311b5550e48a" containerName="istio-proxy"
Oct 30 04:17:27.161009 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.160922    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="425e4e27-91f5-47b0-b8cf-311b5550e48a" containerName="istio-proxy"
Oct 30 04:17:27.161158 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.161042    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="425e4e27-91f5-47b0-b8cf-311b5550e48a" containerName="istio-proxy"
Oct 30 04:17:27.161158 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.161054    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="425e4e27-91f5-47b0-b8cf-311b5550e48a" containerName="istio-proxy"
Oct 30 04:17:27.163759 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.163732    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.178083 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.178063    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-36bbac-openshift-default-dockercfg-n6t7k\""
Oct 30 04:17:27.191987 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.191960    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"]
Oct 30 04:17:27.327462 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.327427    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-workload-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.327610 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.327484    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-credential-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.327610 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.327573    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-data\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.327696 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.327611    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-envoy\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.327696 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.327678    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-podinfo\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.327761 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.327722    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-workload-certs\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.327761 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.327747    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-token\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.327822 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.327766    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zxgf\" (UniqueName: \"kubernetes.io/projected/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-kube-api-access-6zxgf\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.327869 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.327847    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istiod-ca-cert\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.428461 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.428382    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istiod-ca-cert\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.428461 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.428446    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-workload-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.428633 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.428475    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-credential-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.428633 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.428499    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-data\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.428729 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.428630    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-envoy\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.428729 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.428683    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-podinfo\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.428729 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.428714    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-workload-certs\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.428849 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.428752    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-token\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.428902 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.428889    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-data\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.428955 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.428931    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-workload-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.429015 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.428935    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zxgf\" (UniqueName: \"kubernetes.io/projected/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-kube-api-access-6zxgf\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.429126 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.429104    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-workload-certs\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.429211 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.429189    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-credential-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.429265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.429189    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istiod-ca-cert\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.430953 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.430936    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-envoy\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.431117 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.431099    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-podinfo\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.469791 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.469765    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"
Oct 30 04:17:27.470231 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.469801    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb" event={"ID":"425e4e27-91f5-47b0-b8cf-311b5550e48a","Type":"ContainerDied","Data":"471821d323941e5cfe92e229fc3810316776c78f70a155ac71f95accc65b1a22"}
Oct 30 04:17:27.470231 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.469855    4874 scope.go:122] "RemoveContainer" containerID="739fab602a02e6c6af1d1e23b5a9826a98f02b427e58769a710c99ca5ba7dfae"
Oct 30 04:17:27.471197 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.471013    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh" event={"ID":"e1774f86-5e25-4358-b250-d26c6ea6c601","Type":"ContainerStarted","Data":"b4212c87e6403b78797fc3bb48bc0f90ec2c30e1467a6fae587c32943bc6ba14"}
Oct 30 04:17:27.472228 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.472201    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7" event={"ID":"5b386201-0f5e-4add-98a8-bb3bcf476089","Type":"ContainerStarted","Data":"0d6a8888e43b048b4fa2ff7068c549bc39ff94b1185c610485e85f811b369478"}
Oct 30 04:17:27.478534 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.478514    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-token\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.478585 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.478567    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zxgf\" (UniqueName: \"kubernetes.io/projected/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-kube-api-access-6zxgf\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.555816 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.555782    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"]
Oct 30 04:17:27.561171 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.561144    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-de5733f9-openshift-default-79d8fd4dcc-hpkdb"]
Oct 30 04:17:27.776153 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.775518    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:27.943273 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.942316    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj"]
Oct 30 04:17:27.945739 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.945173    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj"
Oct 30 04:17:27.961913 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.961892    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-b4708a\""
Oct 30 04:17:27.966459 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:27.966401    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj"]
Oct 30 04:17:28.051121 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.050825    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a54bdfac-68a5-4f09-bc41-fd51c3209245-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj\" (UID: \"a54bdfac-68a5-4f09-bc41-fd51c3209245\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj"
Oct 30 04:17:28.051121 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.050925    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bsf6\" (UniqueName: \"kubernetes.io/projected/a54bdfac-68a5-4f09-bc41-fd51c3209245-kube-api-access-9bsf6\") pod \"payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj\" (UID: \"a54bdfac-68a5-4f09-bc41-fd51c3209245\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj"
Oct 30 04:17:28.092215 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.091532    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4"]
Oct 30 04:17:28.095045 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.095022    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4"
Oct 30 04:17:28.119879 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.119503    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4"]
Oct 30 04:17:28.133624 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.133596    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-5d74b9-serving-cert\""
Oct 30 04:17:28.134012 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.133989    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"]
Oct 30 04:17:28.147475 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.147424    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:28.147572 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.147504    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:28.147572 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.147548    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:28.152601 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.151648    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a54bdfac-68a5-4f09-bc41-fd51c3209245-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj\" (UID: \"a54bdfac-68a5-4f09-bc41-fd51c3209245\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj"
Oct 30 04:17:28.152601 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.151727    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bsf6\" (UniqueName: \"kubernetes.io/projected/a54bdfac-68a5-4f09-bc41-fd51c3209245-kube-api-access-9bsf6\") pod \"payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj\" (UID: \"a54bdfac-68a5-4f09-bc41-fd51c3209245\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj"
Oct 30 04:17:28.152927 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.152877    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a54bdfac-68a5-4f09-bc41-fd51c3209245-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj\" (UID: \"a54bdfac-68a5-4f09-bc41-fd51c3209245\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj"
Oct 30 04:17:28.180222 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.180194    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bsf6\" (UniqueName: \"kubernetes.io/projected/a54bdfac-68a5-4f09-bc41-fd51c3209245-kube-api-access-9bsf6\") pod \"payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj\" (UID: \"a54bdfac-68a5-4f09-bc41-fd51c3209245\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj"
Oct 30 04:17:28.253287 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.253061    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea64a87b-85b1-46fa-9c51-4e45543bf808-maas-api-tls\") pod \"maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4\" (UID: \"ea64a87b-85b1-46fa-9c51-4e45543bf808\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4"
Oct 30 04:17:28.253287 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.253127    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kmmd\" (UniqueName: \"kubernetes.io/projected/ea64a87b-85b1-46fa-9c51-4e45543bf808-kube-api-access-6kmmd\") pod \"maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4\" (UID: \"ea64a87b-85b1-46fa-9c51-4e45543bf808\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4"
Oct 30 04:17:28.253287 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.253194    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea64a87b-85b1-46fa-9c51-4e45543bf808-metrics-certs\") pod \"maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4\" (UID: \"ea64a87b-85b1-46fa-9c51-4e45543bf808\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4"
Oct 30 04:17:28.253539 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.253392    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea64a87b-85b1-46fa-9c51-4e45543bf808-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4\" (UID: \"ea64a87b-85b1-46fa-9c51-4e45543bf808\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4"
Oct 30 04:17:28.260700 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.260494    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj"
Oct 30 04:17:28.357890 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.356781    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea64a87b-85b1-46fa-9c51-4e45543bf808-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4\" (UID: \"ea64a87b-85b1-46fa-9c51-4e45543bf808\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4"
Oct 30 04:17:28.357890 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.356870    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea64a87b-85b1-46fa-9c51-4e45543bf808-maas-api-tls\") pod \"maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4\" (UID: \"ea64a87b-85b1-46fa-9c51-4e45543bf808\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4"
Oct 30 04:17:28.357890 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.356916    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kmmd\" (UniqueName: \"kubernetes.io/projected/ea64a87b-85b1-46fa-9c51-4e45543bf808-kube-api-access-6kmmd\") pod \"maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4\" (UID: \"ea64a87b-85b1-46fa-9c51-4e45543bf808\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4"
Oct 30 04:17:28.357890 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.356951    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea64a87b-85b1-46fa-9c51-4e45543bf808-metrics-certs\") pod \"maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4\" (UID: \"ea64a87b-85b1-46fa-9c51-4e45543bf808\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4"
Oct 30 04:17:28.358944 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.358916    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea64a87b-85b1-46fa-9c51-4e45543bf808-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4\" (UID: \"ea64a87b-85b1-46fa-9c51-4e45543bf808\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4"
Oct 30 04:17:28.360105 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.359996    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea64a87b-85b1-46fa-9c51-4e45543bf808-metrics-certs\") pod \"maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4\" (UID: \"ea64a87b-85b1-46fa-9c51-4e45543bf808\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4"
Oct 30 04:17:28.363150 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.363090    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea64a87b-85b1-46fa-9c51-4e45543bf808-maas-api-tls\") pod \"maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4\" (UID: \"ea64a87b-85b1-46fa-9c51-4e45543bf808\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4"
Oct 30 04:17:28.403738 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.403694    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kmmd\" (UniqueName: \"kubernetes.io/projected/ea64a87b-85b1-46fa-9c51-4e45543bf808-kube-api-access-6kmmd\") pod \"maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4\" (UID: \"ea64a87b-85b1-46fa-9c51-4e45543bf808\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4"
Oct 30 04:17:28.429529 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.429214    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4"
Oct 30 04:17:28.482218 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.482173    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq" event={"ID":"b3e828ca-ccfa-471b-a2ee-4799f2edb09c","Type":"ContainerStarted","Data":"e815dc04d4160bc30860e7e30ee770dc0b935e3de0039ed90e429ced7e352d26"}
Oct 30 04:17:28.482657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.482273    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq" event={"ID":"b3e828ca-ccfa-471b-a2ee-4799f2edb09c","Type":"ContainerStarted","Data":"c4ff8700b96d3a1734f3020215170e55811f90aded23231388653d79f3a33b84"}
Oct 30 04:17:28.533541 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.533482    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq" podStartSLOduration=1.533461551 podStartE2EDuration="1.533461551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:28.53339991 +0000 UTC m=+1167.945795506" watchObservedRunningTime="2026-10-30 04:17:28.533461551 +0000 UTC m=+1167.945857155"
Oct 30 04:17:28.617360 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.617275    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj"]
Oct 30 04:17:28.623516 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:17:28.623466    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54bdfac_68a5_4f09_bc41_fd51c3209245.slice/crio-fd5d5302a407f7cfe093993dd91ce9a27159933e6d41718e12088616606bc140 WatchSource:0}: Error finding container fd5d5302a407f7cfe093993dd91ce9a27159933e6d41718e12088616606bc140: Status 404 returned error can't find the container with id fd5d5302a407f7cfe093993dd91ce9a27159933e6d41718e12088616606bc140
Oct 30 04:17:28.729769 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.729357    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="425e4e27-91f5-47b0-b8cf-311b5550e48a" path="/var/lib/kubelet/pods/425e4e27-91f5-47b0-b8cf-311b5550e48a/volumes"
Oct 30 04:17:28.775868 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.775834    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:28.784719 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:28.784686    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4"]
Oct 30 04:17:29.505599 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:29.505473    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4" event={"ID":"ea64a87b-85b1-46fa-9c51-4e45543bf808","Type":"ContainerStarted","Data":"495dbb28cd4ac9b43fe51c3e9b2062aff29ac3ddb654c314fc0d00b0124765c6"}
Oct 30 04:17:29.505599 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:29.505537    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4" event={"ID":"ea64a87b-85b1-46fa-9c51-4e45543bf808","Type":"ContainerStarted","Data":"f4d98b94f1feee06b6ac9f579a2c08e04be9cbfc45fc13580f79d4cd318cc92b"}
Oct 30 04:17:29.505599 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:29.505566    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4"
Oct 30 04:17:29.507909 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:29.507840    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj" event={"ID":"a54bdfac-68a5-4f09-bc41-fd51c3209245","Type":"ContainerStarted","Data":"fd5d5302a407f7cfe093993dd91ce9a27159933e6d41718e12088616606bc140"}
Oct 30 04:17:29.548256 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:29.548187    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4" podStartSLOduration=1.240378553 podStartE2EDuration="1.548170315s" totalImagesPullingTime="307.791762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:29.54571432 +0000 UTC m=+1168.958109921" watchObservedRunningTime="2026-10-30 04:17:29.548170315 +0000 UTC m=+1168.960565887"
Oct 30 04:17:29.782428 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:29.782354    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:30.271068 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.271040    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268"]
Oct 30 04:17:30.273504 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.273487    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268"
Oct 30 04:17:30.284743 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.284717    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/245deabe-bbd0-42ef-b7dd-c264c38acdb8-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268\" (UID: \"245deabe-bbd0-42ef-b7dd-c264c38acdb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268"
Oct 30 04:17:30.284849 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.284830    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/245deabe-bbd0-42ef-b7dd-c264c38acdb8-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268\" (UID: \"245deabe-bbd0-42ef-b7dd-c264c38acdb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268"
Oct 30 04:17:30.284892 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.284879    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k87lc\" (UniqueName: \"kubernetes.io/projected/245deabe-bbd0-42ef-b7dd-c264c38acdb8-kube-api-access-k87lc\") pod \"maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268\" (UID: \"245deabe-bbd0-42ef-b7dd-c264c38acdb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268"
Oct 30 04:17:30.284942 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.284908    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/245deabe-bbd0-42ef-b7dd-c264c38acdb8-metrics-certs\") pod \"maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268\" (UID: \"245deabe-bbd0-42ef-b7dd-c264c38acdb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268"
Oct 30 04:17:30.288304 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.288286    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-b4708a-serving-cert\""
Oct 30 04:17:30.296694 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.296671    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268"]
Oct 30 04:17:30.385558 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.385525    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k87lc\" (UniqueName: \"kubernetes.io/projected/245deabe-bbd0-42ef-b7dd-c264c38acdb8-kube-api-access-k87lc\") pod \"maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268\" (UID: \"245deabe-bbd0-42ef-b7dd-c264c38acdb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268"
Oct 30 04:17:30.385687 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.385569    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/245deabe-bbd0-42ef-b7dd-c264c38acdb8-metrics-certs\") pod \"maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268\" (UID: \"245deabe-bbd0-42ef-b7dd-c264c38acdb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268"
Oct 30 04:17:30.385687 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.385609    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/245deabe-bbd0-42ef-b7dd-c264c38acdb8-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268\" (UID: \"245deabe-bbd0-42ef-b7dd-c264c38acdb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268"
Oct 30 04:17:30.385687 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.385668    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/245deabe-bbd0-42ef-b7dd-c264c38acdb8-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268\" (UID: \"245deabe-bbd0-42ef-b7dd-c264c38acdb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268"
Oct 30 04:17:30.385809 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:17:30.385744    4874 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-serving-cert: secret "maas-api-e2e-worker-w3-b4708a-serving-cert" not found
Oct 30 04:17:30.385809 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:17:30.385802    4874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245deabe-bbd0-42ef-b7dd-c264c38acdb8-maas-api-tls podName:245deabe-bbd0-42ef-b7dd-c264c38acdb8 nodeName:}" failed. No retries permitted until 2026-10-30 04:17:30.885787979 +0000 UTC m=+1170.298183531 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/245deabe-bbd0-42ef-b7dd-c264c38acdb8-maas-api-tls") pod "maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268" (UID: "245deabe-bbd0-42ef-b7dd-c264c38acdb8") : secret "maas-api-e2e-worker-w3-b4708a-serving-cert" not found
Oct 30 04:17:30.386366 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.386345    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/245deabe-bbd0-42ef-b7dd-c264c38acdb8-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268\" (UID: \"245deabe-bbd0-42ef-b7dd-c264c38acdb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268"
Oct 30 04:17:30.388034 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.388012    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/245deabe-bbd0-42ef-b7dd-c264c38acdb8-metrics-certs\") pod \"maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268\" (UID: \"245deabe-bbd0-42ef-b7dd-c264c38acdb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268"
Oct 30 04:17:30.415895 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.415868    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k87lc\" (UniqueName: \"kubernetes.io/projected/245deabe-bbd0-42ef-b7dd-c264c38acdb8-kube-api-access-k87lc\") pod \"maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268\" (UID: \"245deabe-bbd0-42ef-b7dd-c264c38acdb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268"
Oct 30 04:17:30.514080 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.514042    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj" event={"ID":"a54bdfac-68a5-4f09-bc41-fd51c3209245","Type":"ContainerStarted","Data":"b96590df5c160a030e0e964170d48f48c760a99a795200d0b79f114009dc7e7c"}
Oct 30 04:17:30.514569 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.514109    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj"
Oct 30 04:17:30.516403 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.516354    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh" event={"ID":"e1774f86-5e25-4358-b250-d26c6ea6c601","Type":"ContainerStarted","Data":"84b66fd27a4a954920fa49279432373a78222a97d62a58de6e3cc8d9fa019334"}
Oct 30 04:17:30.516541 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.516419    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh"
Oct 30 04:17:30.535426 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.535314    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7" event={"ID":"5b386201-0f5e-4add-98a8-bb3bcf476089","Type":"ContainerStarted","Data":"a5fe5dc9d0a8e870cbec1fdba0fb588052c0309b0dfc2a6cdb3de66754e64cf6"}
Oct 30 04:17:30.536841 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.536810    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:30.536983 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.536970    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7"
Oct 30 04:17:30.537706 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.537686    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:30.572562 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.572500    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj" podStartSLOduration=2.12640912 podStartE2EDuration="3.572479885s" totalImagesPullingTime="1.446070765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:30.57067874 +0000 UTC m=+1169.983074314" watchObservedRunningTime="2026-10-30 04:17:30.572479885 +0000 UTC m=+1169.984875461"
Oct 30 04:17:30.612163 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.612116    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh" podStartSLOduration=1.590200485 podStartE2EDuration="4.612104489s" totalImagesPullingTime="3.021904004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:30.611638927 +0000 UTC m=+1170.024034501" watchObservedRunningTime="2026-10-30 04:17:30.612104489 +0000 UTC m=+1170.024500061"
Oct 30 04:17:30.694299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.694180    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7" podStartSLOduration=1.673421549 podStartE2EDuration="4.694162176s" totalImagesPullingTime="3.020740627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:30.692479784 +0000 UTC m=+1170.104875356" watchObservedRunningTime="2026-10-30 04:17:30.694162176 +0000 UTC m=+1170.106557751"
Oct 30 04:17:30.889980 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.889946    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/245deabe-bbd0-42ef-b7dd-c264c38acdb8-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268\" (UID: \"245deabe-bbd0-42ef-b7dd-c264c38acdb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268"
Oct 30 04:17:30.892360 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:30.892337    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/245deabe-bbd0-42ef-b7dd-c264c38acdb8-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268\" (UID: \"245deabe-bbd0-42ef-b7dd-c264c38acdb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268"
Oct 30 04:17:31.184634 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:31.184597    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268"
Oct 30 04:17:31.489343 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:31.489312    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268"]
Oct 30 04:17:31.493009 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:17:31.492977    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245deabe_bbd0_42ef_b7dd_c264c38acdb8.slice/crio-ba85d12cffa40005dea1c2a40b7ad266e1e4768de63affce8a694a9ba0f7b908 WatchSource:0}: Error finding container ba85d12cffa40005dea1c2a40b7ad266e1e4768de63affce8a694a9ba0f7b908: Status 404 returned error can't find the container with id ba85d12cffa40005dea1c2a40b7ad266e1e4768de63affce8a694a9ba0f7b908
Oct 30 04:17:31.540406 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:31.540374    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268" event={"ID":"245deabe-bbd0-42ef-b7dd-c264c38acdb8","Type":"ContainerStarted","Data":"ba85d12cffa40005dea1c2a40b7ad266e1e4768de63affce8a694a9ba0f7b908"}
Oct 30 04:17:32.547556 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:32.547512    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268" event={"ID":"245deabe-bbd0-42ef-b7dd-c264c38acdb8","Type":"ContainerStarted","Data":"04abf9fd437529d169294f5bdedc7966d9fe7455729e77601810322692586378"}
Oct 30 04:17:32.547967 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:32.547614    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268"
Oct 30 04:17:32.589131 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:32.589085    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268" podStartSLOduration=2.329856845 podStartE2EDuration="2.589072694s" totalImagesPullingTime="259.215849ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:32.588908489 +0000 UTC m=+1172.001304087" watchObservedRunningTime="2026-10-30 04:17:32.589072694 +0000 UTC m=+1172.001468263"
Oct 30 04:17:33.994079 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:33.994041    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh"]
Oct 30 04:17:33.997766 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:33.997742    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh"
Oct 30 04:17:34.011380 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:34.011356    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-a2f984-serving-cert\""
Oct 30 04:17:34.025310 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:34.025286    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh"]
Oct 30 04:17:34.032791 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:34.032754    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7479a6e1-f414-411e-b1a0-a56fe33f8262-metrics-certs\") pod \"maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh\" (UID: \"7479a6e1-f414-411e-b1a0-a56fe33f8262\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh"
Oct 30 04:17:34.032917 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:34.032826    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7479a6e1-f414-411e-b1a0-a56fe33f8262-maas-api-tls\") pod \"maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh\" (UID: \"7479a6e1-f414-411e-b1a0-a56fe33f8262\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh"
Oct 30 04:17:34.032917 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:34.032865    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7479a6e1-f414-411e-b1a0-a56fe33f8262-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh\" (UID: \"7479a6e1-f414-411e-b1a0-a56fe33f8262\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh"
Oct 30 04:17:34.033028 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:34.032924    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nh5b\" (UniqueName: \"kubernetes.io/projected/7479a6e1-f414-411e-b1a0-a56fe33f8262-kube-api-access-6nh5b\") pod \"maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh\" (UID: \"7479a6e1-f414-411e-b1a0-a56fe33f8262\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh"
Oct 30 04:17:34.134163 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:34.134132    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nh5b\" (UniqueName: \"kubernetes.io/projected/7479a6e1-f414-411e-b1a0-a56fe33f8262-kube-api-access-6nh5b\") pod \"maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh\" (UID: \"7479a6e1-f414-411e-b1a0-a56fe33f8262\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh"
Oct 30 04:17:34.134359 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:34.134182    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7479a6e1-f414-411e-b1a0-a56fe33f8262-metrics-certs\") pod \"maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh\" (UID: \"7479a6e1-f414-411e-b1a0-a56fe33f8262\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh"
Oct 30 04:17:34.134359 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:34.134228    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7479a6e1-f414-411e-b1a0-a56fe33f8262-maas-api-tls\") pod \"maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh\" (UID: \"7479a6e1-f414-411e-b1a0-a56fe33f8262\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh"
Oct 30 04:17:34.134359 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:34.134279    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7479a6e1-f414-411e-b1a0-a56fe33f8262-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh\" (UID: \"7479a6e1-f414-411e-b1a0-a56fe33f8262\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh"
Oct 30 04:17:34.134524 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:17:34.134413    4874 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-serving-cert: secret "maas-api-e2e-worker-w4-a2f984-serving-cert" not found
Oct 30 04:17:34.134524 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:17:34.134494    4874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7479a6e1-f414-411e-b1a0-a56fe33f8262-maas-api-tls podName:7479a6e1-f414-411e-b1a0-a56fe33f8262 nodeName:}" failed. No retries permitted until 2026-10-30 04:17:34.634477814 +0000 UTC m=+1174.046873369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7479a6e1-f414-411e-b1a0-a56fe33f8262-maas-api-tls") pod "maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh" (UID: "7479a6e1-f414-411e-b1a0-a56fe33f8262") : secret "maas-api-e2e-worker-w4-a2f984-serving-cert" not found
Oct 30 04:17:34.134938 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:34.134918    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7479a6e1-f414-411e-b1a0-a56fe33f8262-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh\" (UID: \"7479a6e1-f414-411e-b1a0-a56fe33f8262\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh"
Oct 30 04:17:34.136593 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:34.136562    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7479a6e1-f414-411e-b1a0-a56fe33f8262-metrics-certs\") pod \"maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh\" (UID: \"7479a6e1-f414-411e-b1a0-a56fe33f8262\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh"
Oct 30 04:17:34.157393 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:34.157369    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nh5b\" (UniqueName: \"kubernetes.io/projected/7479a6e1-f414-411e-b1a0-a56fe33f8262-kube-api-access-6nh5b\") pod \"maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh\" (UID: \"7479a6e1-f414-411e-b1a0-a56fe33f8262\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh"
Oct 30 04:17:34.641755 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:34.641714    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7479a6e1-f414-411e-b1a0-a56fe33f8262-maas-api-tls\") pod \"maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh\" (UID: \"7479a6e1-f414-411e-b1a0-a56fe33f8262\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh"
Oct 30 04:17:34.644044 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:34.644024    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7479a6e1-f414-411e-b1a0-a56fe33f8262-maas-api-tls\") pod \"maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh\" (UID: \"7479a6e1-f414-411e-b1a0-a56fe33f8262\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh"
Oct 30 04:17:34.910657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:34.910579    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh"
Oct 30 04:17:35.225060 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:35.225039    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh"]
Oct 30 04:17:35.228107 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:17:35.228078    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7479a6e1_f414_411e_b1a0_a56fe33f8262.slice/crio-39315431dbc66f22df94027dfdfa962675cb5558bfa7fbb37972d96c5ac34773 WatchSource:0}: Error finding container 39315431dbc66f22df94027dfdfa962675cb5558bfa7fbb37972d96c5ac34773: Status 404 returned error can't find the container with id 39315431dbc66f22df94027dfdfa962675cb5558bfa7fbb37972d96c5ac34773
Oct 30 04:17:35.537335 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:35.537257    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h"]
Oct 30 04:17:35.537478 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:35.537450    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h" podUID="7001f753-082c-4759-b6e4-689d1d512d68" containerName="maas-api" containerID="cri-o://90236b2a5e7952c4e111f31fedd0c919eeff2d076771066825574283e167675e" gracePeriod=30
Oct 30 04:17:35.542313 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:35.542281    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4"
Oct 30 04:17:35.599533 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:35.599494    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh" event={"ID":"7479a6e1-f414-411e-b1a0-a56fe33f8262","Type":"ContainerStarted","Data":"39315431dbc66f22df94027dfdfa962675cb5558bfa7fbb37972d96c5ac34773"}
Oct 30 04:17:35.913358 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:35.913331    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h"
Oct 30 04:17:35.963264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:35.959566    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7001f753-082c-4759-b6e4-689d1d512d68-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7001f753-082c-4759-b6e4-689d1d512d68-maas-api-tls\") pod \"7001f753-082c-4759-b6e4-689d1d512d68\" (UID: \"7001f753-082c-4759-b6e4-689d1d512d68\") "
Oct 30 04:17:35.963264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:35.959649    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7001f753-082c-4759-b6e4-689d1d512d68-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7001f753-082c-4759-b6e4-689d1d512d68-metrics-certs\") pod \"7001f753-082c-4759-b6e4-689d1d512d68\" (UID: \"7001f753-082c-4759-b6e4-689d1d512d68\") "
Oct 30 04:17:35.963264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:35.959687    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7001f753-082c-4759-b6e4-689d1d512d68-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7001f753-082c-4759-b6e4-689d1d512d68-openshift-service-ca\") pod \"7001f753-082c-4759-b6e4-689d1d512d68\" (UID: \"7001f753-082c-4759-b6e4-689d1d512d68\") "
Oct 30 04:17:35.963264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:35.959807    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7001f753-082c-4759-b6e4-689d1d512d68-kube-api-access-7gmsc\" (UniqueName: \"kubernetes.io/projected/7001f753-082c-4759-b6e4-689d1d512d68-kube-api-access-7gmsc\") pod \"7001f753-082c-4759-b6e4-689d1d512d68\" (UID: \"7001f753-082c-4759-b6e4-689d1d512d68\") "
Oct 30 04:17:35.963264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:35.960608    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7001f753-082c-4759-b6e4-689d1d512d68-openshift-service-ca" pod "7001f753-082c-4759-b6e4-689d1d512d68" (UID: "7001f753-082c-4759-b6e4-689d1d512d68"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:35.966221 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:35.966160    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7001f753-082c-4759-b6e4-689d1d512d68-metrics-certs" pod "7001f753-082c-4759-b6e4-689d1d512d68" (UID: "7001f753-082c-4759-b6e4-689d1d512d68"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:35.966559 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:35.966515    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7001f753-082c-4759-b6e4-689d1d512d68-maas-api-tls" pod "7001f753-082c-4759-b6e4-689d1d512d68" (UID: "7001f753-082c-4759-b6e4-689d1d512d68"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:35.975765 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:35.975738    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7001f753-082c-4759-b6e4-689d1d512d68-kube-api-access-7gmsc" pod "7001f753-082c-4759-b6e4-689d1d512d68" (UID: "7001f753-082c-4759-b6e4-689d1d512d68"). InnerVolumeSpecName "kube-api-access-7gmsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:36.061551 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:36.061521    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7001f753-082c-4759-b6e4-689d1d512d68-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:36.061551 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:36.061548    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7001f753-082c-4759-b6e4-689d1d512d68-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:36.061551 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:36.061557    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7001f753-082c-4759-b6e4-689d1d512d68-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:36.061775 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:36.061566    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gmsc\" (UniqueName: \"kubernetes.io/projected/7001f753-082c-4759-b6e4-689d1d512d68-kube-api-access-7gmsc\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:17:36.605547 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:36.605448    4874 generic.go:356] "Generic (PLEG): container finished" podID="7001f753-082c-4759-b6e4-689d1d512d68" containerID="90236b2a5e7952c4e111f31fedd0c919eeff2d076771066825574283e167675e" exitCode=0
Oct 30 04:17:36.605547 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:36.605502    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h"
Oct 30 04:17:36.606051 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:36.605577    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h" event={"ID":"7001f753-082c-4759-b6e4-689d1d512d68","Type":"ContainerDied","Data":"90236b2a5e7952c4e111f31fedd0c919eeff2d076771066825574283e167675e"}
Oct 30 04:17:36.606051 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:36.605634    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h" event={"ID":"7001f753-082c-4759-b6e4-689d1d512d68","Type":"ContainerDied","Data":"5398cedbfab5e913de6df58d3733c89a38dada5975ca8cfad7f6d08aa22f75a2"}
Oct 30 04:17:36.606051 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:36.605668    4874 scope.go:122] "RemoveContainer" containerID="90236b2a5e7952c4e111f31fedd0c919eeff2d076771066825574283e167675e"
Oct 30 04:17:36.607623 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:36.607603    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh" event={"ID":"7479a6e1-f414-411e-b1a0-a56fe33f8262","Type":"ContainerStarted","Data":"a2bbc54cb3fbf6aed2870e3a6214f89e95f7a25618d0f2a1c8e36199c1f8fa59"}
Oct 30 04:17:36.607755 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:36.607733    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh"
Oct 30 04:17:36.615732 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:36.615714    4874 scope.go:122] "RemoveContainer" containerID="90236b2a5e7952c4e111f31fedd0c919eeff2d076771066825574283e167675e"
Oct 30 04:17:36.615969 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:17:36.615950    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90236b2a5e7952c4e111f31fedd0c919eeff2d076771066825574283e167675e\": container with ID starting with 90236b2a5e7952c4e111f31fedd0c919eeff2d076771066825574283e167675e not found: ID does not exist" containerID="90236b2a5e7952c4e111f31fedd0c919eeff2d076771066825574283e167675e"
Oct 30 04:17:36.616029 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:36.615976    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90236b2a5e7952c4e111f31fedd0c919eeff2d076771066825574283e167675e"} err="failed to get container status \"90236b2a5e7952c4e111f31fedd0c919eeff2d076771066825574283e167675e\": rpc error: code = NotFound desc = could not find container \"90236b2a5e7952c4e111f31fedd0c919eeff2d076771066825574283e167675e\": container with ID starting with 90236b2a5e7952c4e111f31fedd0c919eeff2d076771066825574283e167675e not found: ID does not exist"
Oct 30 04:17:36.660847 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:36.660811    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h"]
Oct 30 04:17:36.662277 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:36.662249    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cb1a18be-89c57dd45-lq77h"]
Oct 30 04:17:36.713708 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:36.713648    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh" podStartSLOduration=3.343403994 podStartE2EDuration="3.71350846s" totalImagesPullingTime="370.104466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:36.711832324 +0000 UTC m=+1176.124227896" watchObservedRunningTime="2026-10-30 04:17:36.71350846 +0000 UTC m=+1176.125904034"
Oct 30 04:17:36.727788 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:36.727757    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7001f753-082c-4759-b6e4-689d1d512d68" path="/var/lib/kubelet/pods/7001f753-082c-4759-b6e4-689d1d512d68/volumes"
Oct 30 04:17:37.146424 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:37.146382    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:17:38.084508 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:38.081223    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd"]
Oct 30 04:17:38.084508 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:38.081845    4874 cpu_manager.go:414] "containerMap: removing container" podUID="7001f753-082c-4759-b6e4-689d1d512d68" containerName="maas-api"
Oct 30 04:17:38.084508 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:38.081865    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7001f753-082c-4759-b6e4-689d1d512d68" containerName="maas-api"
Oct 30 04:17:38.084508 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:38.082002    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="7001f753-082c-4759-b6e4-689d1d512d68" containerName="maas-api"
Oct 30 04:17:38.088015 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:38.087990    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd"
Oct 30 04:17:38.106274 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:38.105369    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd"]
Oct 30 04:17:38.116260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:38.115888    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-maas-api-tls\") pod \"maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd\" (UID: \"96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd"
Oct 30 04:17:38.116260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:38.115937    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-metrics-certs\") pod \"maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd\" (UID: \"96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd"
Oct 30 04:17:38.116260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:38.115967    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw924\" (UniqueName: \"kubernetes.io/projected/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-kube-api-access-vw924\") pod \"maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd\" (UID: \"96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd"
Oct 30 04:17:38.116260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:38.116053    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd\" (UID: \"96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd"
Oct 30 04:17:38.129442 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:38.129415    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-36bbac-serving-cert\""
Oct 30 04:17:38.218890 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:38.218710    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd\" (UID: \"96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd"
Oct 30 04:17:38.218890 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:38.218795    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-maas-api-tls\") pod \"maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd\" (UID: \"96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd"
Oct 30 04:17:38.218890 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:38.218811    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-metrics-certs\") pod \"maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd\" (UID: \"96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd"
Oct 30 04:17:38.218890 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:38.218829    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw924\" (UniqueName: \"kubernetes.io/projected/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-kube-api-access-vw924\") pod \"maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd\" (UID: \"96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd"
Oct 30 04:17:38.219687 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:38.219658    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd\" (UID: \"96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd"
Oct 30 04:17:38.222441 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:38.222418    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-maas-api-tls\") pod \"maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd\" (UID: \"96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd"
Oct 30 04:17:38.222636 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:38.222615    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-metrics-certs\") pod \"maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd\" (UID: \"96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd"
Oct 30 04:17:38.241137 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:38.241113    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw924\" (UniqueName: \"kubernetes.io/projected/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-kube-api-access-vw924\") pod \"maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd\" (UID: \"96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd"
Oct 30 04:17:38.432319 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:38.432170    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd"
Oct 30 04:17:38.559706 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:38.559681    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268"
Oct 30 04:17:38.756224 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:38.756195    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd"]
Oct 30 04:17:38.759737 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:17:38.759710    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96eb0e3a_5bb9_45ee_ac86_80c7a0cb9c08.slice/crio-839ab1e346a0a66b83b433dd80d3d53e85a6f3747174a95b35549d0245413558 WatchSource:0}: Error finding container 839ab1e346a0a66b83b433dd80d3d53e85a6f3747174a95b35549d0245413558: Status 404 returned error can't find the container with id 839ab1e346a0a66b83b433dd80d3d53e85a6f3747174a95b35549d0245413558
Oct 30 04:17:38.761203 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:38.761183    4874 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 04:17:39.627183 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:39.627086    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd" event={"ID":"96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08","Type":"ContainerStarted","Data":"e1c24733c803a9b80e10ad3c4166a08a0099863d7503ea3ceeacea09cf87db3d"}
Oct 30 04:17:39.627183 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:39.627138    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd" event={"ID":"96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08","Type":"ContainerStarted","Data":"839ab1e346a0a66b83b433dd80d3d53e85a6f3747174a95b35549d0245413558"}
Oct 30 04:17:39.627716 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:39.627198    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd"
Oct 30 04:17:41.543570 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:41.543541    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj"
Oct 30 04:17:41.543942 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:41.543610    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh"
Oct 30 04:17:41.590262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:41.586449    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd" podStartSLOduration=3.098763334 podStartE2EDuration="3.586429244s" totalImagesPullingTime="487.66591ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:39.687158526 +0000 UTC m=+1179.099554100" watchObservedRunningTime="2026-10-30 04:17:41.586429244 +0000 UTC m=+1180.998824818"
Oct 30 04:17:42.551444 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:42.551370    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7"
Oct 30 04:17:42.620724 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:42.620696    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh"
Oct 30 04:17:42.644481 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:42.644446    4874 generic.go:356] "Generic (PLEG): container finished" podID="10b28e83-cdc5-4591-858b-80df98885d4e" containerID="a3938d264c0c2e896b85ff65153cf0422811442940c3ab85b93c372e65b1cd62" exitCode=0
Oct 30 04:17:42.644658 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:42.644605    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj" event={"ID":"10b28e83-cdc5-4591-858b-80df98885d4e","Type":"ContainerDied","Data":"a3938d264c0c2e896b85ff65153cf0422811442940c3ab85b93c372e65b1cd62"}
Oct 30 04:17:42.644995 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:42.644976    4874 scope.go:122] "RemoveContainer" containerID="a3938d264c0c2e896b85ff65153cf0422811442940c3ab85b93c372e65b1cd62"
Oct 30 04:17:42.646185 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:42.646157    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:42.646308 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:42.646259    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:42.646377 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:42.646311    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:43.651840 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:43.651805    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj" event={"ID":"10b28e83-cdc5-4591-858b-80df98885d4e","Type":"ContainerStarted","Data":"2bb175b2ac9439e18009ac4789561f0ab4f8f4196252ee5ce48c9e0fdc5cfff2"}
Oct 30 04:17:43.960308 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:43.958270    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:17:44.570264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:44.568484    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:44.570264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:44.569064    4874 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 04:17:44.570264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:44.569115    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj" podUID="10b28e83-cdc5-4591-858b-80df98885d4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 04:17:45.571265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:45.568607    4874 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 04:17:45.571265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:45.568671    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj" podUID="10b28e83-cdc5-4591-858b-80df98885d4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 04:17:45.670019 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:45.669988    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd"
Oct 30 04:17:46.568262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:46.567644    4874 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 04:17:46.568262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:46.567705    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj" podUID="10b28e83-cdc5-4591-858b-80df98885d4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 04:17:47.568061 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:47.567627    4874 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 04:17:47.568061 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:47.567690    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj" podUID="10b28e83-cdc5-4591-858b-80df98885d4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 04:17:48.584036 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:48.583598    4874 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 04:17:48.584036 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:48.583660    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj" podUID="10b28e83-cdc5-4591-858b-80df98885d4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 04:17:49.736784 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:49.736753    4874 generic.go:356] "Generic (PLEG): container finished" podID="4e716825-de92-4c9d-9b36-007ea4218a98" containerID="2f09cbdac2cbd04e842944626106cc89e120926fdef07ae3a2074d673c42284d" exitCode=0
Oct 30 04:17:49.737305 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:49.736835    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s" event={"ID":"4e716825-de92-4c9d-9b36-007ea4218a98","Type":"ContainerDied","Data":"2f09cbdac2cbd04e842944626106cc89e120926fdef07ae3a2074d673c42284d"}
Oct 30 04:17:49.737305 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:49.737158    4874 scope.go:122] "RemoveContainer" containerID="2f09cbdac2cbd04e842944626106cc89e120926fdef07ae3a2074d673c42284d"
Oct 30 04:17:49.738714 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:49.738685    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:49.738827 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:49.738748    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:49.738827 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:49.738779    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:50.040794 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:50.040499    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:50.040794 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:50.040783    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:50.042323 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:50.042301    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:17:50.821601 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:50.821555    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s" event={"ID":"4e716825-de92-4c9d-9b36-007ea4218a98","Type":"ContainerStarted","Data":"e184a4f17326316b5f030bfa193458ae7d9a9d0d8aafb46449a52886b23fb315"}
Oct 30 04:17:51.017418 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:51.014441    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:51.017418 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:51.016793    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 04:17:51.017418 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:51.016907    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s" podUID="4e716825-de92-4c9d-9b36-007ea4218a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 04:17:51.658661 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:51.658621    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb"]
Oct 30 04:17:51.696037 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:51.695997    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb"
Oct 30 04:17:51.754127 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:51.754091    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb"]
Oct 30 04:17:51.789866 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:51.789832    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-6796b1-serving-cert\""
Oct 30 04:17:51.794110 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:51.794080    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb\" (UID: \"8d278356-d7d2-42cd-bba4-e5ef1867fa1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb"
Oct 30 04:17:51.794279 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:51.794132    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-metrics-certs\") pod \"maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb\" (UID: \"8d278356-d7d2-42cd-bba4-e5ef1867fa1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb"
Oct 30 04:17:51.794365 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:51.794340    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhkdd\" (UniqueName: \"kubernetes.io/projected/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-kube-api-access-vhkdd\") pod \"maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb\" (UID: \"8d278356-d7d2-42cd-bba4-e5ef1867fa1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb"
Oct 30 04:17:51.794438 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:51.794420    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-maas-api-tls\") pod \"maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb\" (UID: \"8d278356-d7d2-42cd-bba4-e5ef1867fa1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb"
Oct 30 04:17:51.895930 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:51.895894    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb\" (UID: \"8d278356-d7d2-42cd-bba4-e5ef1867fa1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb"
Oct 30 04:17:51.896423 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:51.895954    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-metrics-certs\") pod \"maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb\" (UID: \"8d278356-d7d2-42cd-bba4-e5ef1867fa1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb"
Oct 30 04:17:51.896423 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:51.896075    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhkdd\" (UniqueName: \"kubernetes.io/projected/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-kube-api-access-vhkdd\") pod \"maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb\" (UID: \"8d278356-d7d2-42cd-bba4-e5ef1867fa1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb"
Oct 30 04:17:51.896423 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:51.896148    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-maas-api-tls\") pod \"maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb\" (UID: \"8d278356-d7d2-42cd-bba4-e5ef1867fa1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb"
Oct 30 04:17:51.896739 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:51.896710    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb\" (UID: \"8d278356-d7d2-42cd-bba4-e5ef1867fa1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb"
Oct 30 04:17:51.898682 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:51.898650    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-maas-api-tls\") pod \"maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb\" (UID: \"8d278356-d7d2-42cd-bba4-e5ef1867fa1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb"
Oct 30 04:17:51.898959 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:51.898929    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-metrics-certs\") pod \"maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb\" (UID: \"8d278356-d7d2-42cd-bba4-e5ef1867fa1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb"
Oct 30 04:17:51.919386 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:51.919330    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhkdd\" (UniqueName: \"kubernetes.io/projected/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-kube-api-access-vhkdd\") pod \"maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb\" (UID: \"8d278356-d7d2-42cd-bba4-e5ef1867fa1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb"
Oct 30 04:17:52.011249 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:52.010644    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb"
Oct 30 04:17:52.011918 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:52.011887    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 04:17:52.011986 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:52.011944    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s" podUID="4e716825-de92-4c9d-9b36-007ea4218a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 04:17:52.323909 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:52.323872    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb"]
Oct 30 04:17:52.326957 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:17:52.326924    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d278356_d7d2_42cd_bba4_e5ef1867fa1c.slice/crio-8609e8f0ec7b2a6b97198453485a3bef71d272d1cb5272cbd40dc6f41f74d1c1 WatchSource:0}: Error finding container 8609e8f0ec7b2a6b97198453485a3bef71d272d1cb5272cbd40dc6f41f74d1c1: Status 404 returned error can't find the container with id 8609e8f0ec7b2a6b97198453485a3bef71d272d1cb5272cbd40dc6f41f74d1c1
Oct 30 04:17:52.833025 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:52.832985    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb" event={"ID":"8d278356-d7d2-42cd-bba4-e5ef1867fa1c","Type":"ContainerStarted","Data":"6e9c4196c6c3c1a630fe2804f2dafdc1beae2a555157227f4d74b968f9a0fa40"}
Oct 30 04:17:52.833201 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:52.833052    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb" event={"ID":"8d278356-d7d2-42cd-bba4-e5ef1867fa1c","Type":"ContainerStarted","Data":"8609e8f0ec7b2a6b97198453485a3bef71d272d1cb5272cbd40dc6f41f74d1c1"}
Oct 30 04:17:52.833201 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:52.833106    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb"
Oct 30 04:17:52.892091 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:52.892025    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb" podStartSLOduration=1.564673375 podStartE2EDuration="1.892006905s" totalImagesPullingTime="327.33353ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:52.873492656 +0000 UTC m=+1192.285888282" watchObservedRunningTime="2026-10-30 04:17:52.892006905 +0000 UTC m=+1192.304402480"
Oct 30 04:17:53.017660 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:53.017615    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 04:17:53.018130 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:53.017681    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s" podUID="4e716825-de92-4c9d-9b36-007ea4218a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 04:17:53.439876 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:53.439796    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:54.012301 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:54.012259    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 04:17:54.012480 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:54.012332    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s" podUID="4e716825-de92-4c9d-9b36-007ea4218a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 04:17:54.849989 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:54.849955    4874 generic.go:356] "Generic (PLEG): container finished" podID="b3e828ca-ccfa-471b-a2ee-4799f2edb09c" containerID="e815dc04d4160bc30860e7e30ee770dc0b935e3de0039ed90e429ced7e352d26" exitCode=0
Oct 30 04:17:54.850558 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:54.850030    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq" event={"ID":"b3e828ca-ccfa-471b-a2ee-4799f2edb09c","Type":"ContainerDied","Data":"e815dc04d4160bc30860e7e30ee770dc0b935e3de0039ed90e429ced7e352d26"}
Oct 30 04:17:54.850558 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:54.850502    4874 scope.go:122] "RemoveContainer" containerID="e815dc04d4160bc30860e7e30ee770dc0b935e3de0039ed90e429ced7e352d26"
Oct 30 04:17:54.851803 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:54.851779    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:54.851863 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:54.851842    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:54.851901 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:54.851875    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:17:55.012728 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:55.012693    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 04:17:55.012900 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:55.012775    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s" podUID="4e716825-de92-4c9d-9b36-007ea4218a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 04:17:55.062263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:55.061393    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:17:55.655281 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:17:55.655230    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:17:55.858853 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:55.858822    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq" event={"ID":"b3e828ca-ccfa-471b-a2ee-4799f2edb09c","Type":"ContainerStarted","Data":"a84cc1a728f76def95c124aedbf8f320cba0899d25f57e7adc8129ea27fdaa94"}
Oct 30 04:17:56.181278 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:56.180580    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:56.195265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:56.192478    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:17:56.779299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:56.779040    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:17:56.779299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:56.779182    4874 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 04:17:56.779299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:56.779227    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq" podUID="b3e828ca-ccfa-471b-a2ee-4799f2edb09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 04:17:57.784420 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:57.784385    4874 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 04:17:57.785028 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:57.784995    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq" podUID="b3e828ca-ccfa-471b-a2ee-4799f2edb09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 04:17:58.789630 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:58.789596    4874 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 04:17:58.790048 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:58.789654    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq" podUID="b3e828ca-ccfa-471b-a2ee-4799f2edb09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 04:17:58.849784 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:58.849757    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb"
Oct 30 04:17:59.779579 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:59.779537    4874 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 04:17:59.779757 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:17:59.779597    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq" podUID="b3e828ca-ccfa-471b-a2ee-4799f2edb09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 04:18:00.536880 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:00.536846    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:18:00.789766 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:00.789649    4874 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 04:18:00.789766 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:00.789726    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq" podUID="b3e828ca-ccfa-471b-a2ee-4799f2edb09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 04:18:00.823110 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:00.823086    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 04:18:00.845608 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:00.845495    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 04:18:01.779988 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:01.779958    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:18:01.780860 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:01.780841    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:18:07.985879 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:07.985846    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:18:09.637060 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.637025    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"]
Oct 30 04:18:09.641093 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.641013    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 04:18:09.669280 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.669232    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5d74b9\"/\"premium-w2-5d74b9-kserve-self-signed-certs\""
Oct 30 04:18:09.669497 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.669472    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-5d74b9\"/\"kube-root-ca.crt\""
Oct 30 04:18:09.669497 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.669292    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5d74b9\"/\"default-dockercfg-j7cqj\""
Oct 30 04:18:09.669679 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.669318    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-5d74b9\"/\"openshift-service-ca.crt\""
Oct 30 04:18:09.671404 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.671383    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"]
Oct 30 04:18:09.773836 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.773807    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-home\") pod \"premium-w2-5d74b9-kserve-76956ddc5c-pdqw7\" (UID: \"1463c013-e615-4142-b799-56b17e67a474\") " pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 04:18:09.774002 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.773879    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-dshm\") pod \"premium-w2-5d74b9-kserve-76956ddc5c-pdqw7\" (UID: \"1463c013-e615-4142-b799-56b17e67a474\") " pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 04:18:09.774002 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.773922    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-tmp-dir\") pod \"premium-w2-5d74b9-kserve-76956ddc5c-pdqw7\" (UID: \"1463c013-e615-4142-b799-56b17e67a474\") " pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 04:18:09.774002 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.773971    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc75q\" (UniqueName: \"kubernetes.io/projected/1463c013-e615-4142-b799-56b17e67a474-kube-api-access-nc75q\") pod \"premium-w2-5d74b9-kserve-76956ddc5c-pdqw7\" (UID: \"1463c013-e615-4142-b799-56b17e67a474\") " pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 04:18:09.774126 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.774083    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-model-cache\") pod \"premium-w2-5d74b9-kserve-76956ddc5c-pdqw7\" (UID: \"1463c013-e615-4142-b799-56b17e67a474\") " pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 04:18:09.774165 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.774129    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1463c013-e615-4142-b799-56b17e67a474-tls-certs\") pod \"premium-w2-5d74b9-kserve-76956ddc5c-pdqw7\" (UID: \"1463c013-e615-4142-b799-56b17e67a474\") " pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 04:18:09.877543 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.877500    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-home\") pod \"premium-w2-5d74b9-kserve-76956ddc5c-pdqw7\" (UID: \"1463c013-e615-4142-b799-56b17e67a474\") " pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 04:18:09.877885 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.877863    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-dshm\") pod \"premium-w2-5d74b9-kserve-76956ddc5c-pdqw7\" (UID: \"1463c013-e615-4142-b799-56b17e67a474\") " pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 04:18:09.879397 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.879370    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-tmp-dir\") pod \"premium-w2-5d74b9-kserve-76956ddc5c-pdqw7\" (UID: \"1463c013-e615-4142-b799-56b17e67a474\") " pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 04:18:09.884595 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.884561    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc75q\" (UniqueName: \"kubernetes.io/projected/1463c013-e615-4142-b799-56b17e67a474-kube-api-access-nc75q\") pod \"premium-w2-5d74b9-kserve-76956ddc5c-pdqw7\" (UID: \"1463c013-e615-4142-b799-56b17e67a474\") " pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 04:18:09.884818 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.884799    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-model-cache\") pod \"premium-w2-5d74b9-kserve-76956ddc5c-pdqw7\" (UID: \"1463c013-e615-4142-b799-56b17e67a474\") " pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 04:18:09.885077 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.885059    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1463c013-e615-4142-b799-56b17e67a474-tls-certs\") pod \"premium-w2-5d74b9-kserve-76956ddc5c-pdqw7\" (UID: \"1463c013-e615-4142-b799-56b17e67a474\") " pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 04:18:09.889754 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.878070    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-home\") pod \"premium-w2-5d74b9-kserve-76956ddc5c-pdqw7\" (UID: \"1463c013-e615-4142-b799-56b17e67a474\") " pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 04:18:09.889754 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.882647    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-tmp-dir\") pod \"premium-w2-5d74b9-kserve-76956ddc5c-pdqw7\" (UID: \"1463c013-e615-4142-b799-56b17e67a474\") " pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 04:18:09.889754 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.886914    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-model-cache\") pod \"premium-w2-5d74b9-kserve-76956ddc5c-pdqw7\" (UID: \"1463c013-e615-4142-b799-56b17e67a474\") " pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 04:18:09.889754 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.889542    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-dshm\") pod \"premium-w2-5d74b9-kserve-76956ddc5c-pdqw7\" (UID: \"1463c013-e615-4142-b799-56b17e67a474\") " pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 04:18:09.890089 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.890070    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1463c013-e615-4142-b799-56b17e67a474-tls-certs\") pod \"premium-w2-5d74b9-kserve-76956ddc5c-pdqw7\" (UID: \"1463c013-e615-4142-b799-56b17e67a474\") " pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 04:18:09.917377 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.917344    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc75q\" (UniqueName: \"kubernetes.io/projected/1463c013-e615-4142-b799-56b17e67a474-kube-api-access-nc75q\") pod \"premium-w2-5d74b9-kserve-76956ddc5c-pdqw7\" (UID: \"1463c013-e615-4142-b799-56b17e67a474\") " pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 04:18:09.952737 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:09.952710    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 04:18:10.253855 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:10.253775    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"]
Oct 30 04:18:10.256473 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:18:10.256447    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1463c013_e615_4142_b799_56b17e67a474.slice/crio-a335dd9a92d28a27f1fb0a184d4d9785b9602e3d2e7b6cd41b20d1354ee54921 WatchSource:0}: Error finding container a335dd9a92d28a27f1fb0a184d4d9785b9602e3d2e7b6cd41b20d1354ee54921: Status 404 returned error can't find the container with id a335dd9a92d28a27f1fb0a184d4d9785b9602e3d2e7b6cd41b20d1354ee54921
Oct 30 04:18:10.965585 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:10.965547    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7" event={"ID":"1463c013-e615-4142-b799-56b17e67a474","Type":"ContainerStarted","Data":"339f649f7b6920cc8a9bb5a59c418959bff8afe37a63d9bb9f4d0e5aa5ee52a6"}
Oct 30 04:18:10.965957 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:10.965595    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7" event={"ID":"1463c013-e615-4142-b799-56b17e67a474","Type":"ContainerStarted","Data":"a335dd9a92d28a27f1fb0a184d4d9785b9602e3d2e7b6cd41b20d1354ee54921"}
Oct 30 04:18:11.005820 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:11.005771    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7" podStartSLOduration=2.005758464 podStartE2EDuration="2.005758464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:11.004905735 +0000 UTC m=+1210.417301308" watchObservedRunningTime="2026-10-30 04:18:11.005758464 +0000 UTC m=+1210.418154039"
Oct 30 04:18:11.953566 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:11.953535    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 04:18:19.953628 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:19.953550    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 04:18:19.965898 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:19.965873    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 04:18:20.021018 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:20.020997    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 04:18:20.142366 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:20.142332    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:18:25.608955 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:18:25.608918    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:18:29.899410 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:29.899377    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"]
Oct 30 04:18:29.904361 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:29.904341    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 04:18:29.918938 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:29.918912    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-36bbac\"/\"premium-w0-36bbac-kserve-self-signed-certs\""
Oct 30 04:18:29.930672 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:29.930650    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-36bbac\"/\"kube-root-ca.crt\""
Oct 30 04:18:29.930672 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:29.930649    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-36bbac\"/\"default-dockercfg-llsp2\""
Oct 30 04:18:29.932830 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:29.932814    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-36bbac\"/\"openshift-service-ca.crt\""
Oct 30 04:18:29.933573 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:29.933551    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"]
Oct 30 04:18:30.029602 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:30.029573    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-home\") pod \"premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9\" (UID: \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\") " pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 04:18:30.029795 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:30.029609    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-dshm\") pod \"premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9\" (UID: \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\") " pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 04:18:30.029795 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:30.029628    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v9hc\" (UniqueName: \"kubernetes.io/projected/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-kube-api-access-7v9hc\") pod \"premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9\" (UID: \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\") " pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 04:18:30.029890 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:30.029787    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-model-cache\") pod \"premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9\" (UID: \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\") " pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 04:18:30.029890 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:30.029872    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-tmp-dir\") pod \"premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9\" (UID: \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\") " pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 04:18:30.029961 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:30.029932    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-tls-certs\") pod \"premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9\" (UID: \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\") " pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 04:18:30.130865 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:30.130806    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-tmp-dir\") pod \"premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9\" (UID: \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\") " pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 04:18:30.131057 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:30.130915    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-tls-certs\") pod \"premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9\" (UID: \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\") " pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 04:18:30.131057 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:30.130976    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-home\") pod \"premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9\" (UID: \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\") " pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 04:18:30.131057 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:30.131006    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-dshm\") pod \"premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9\" (UID: \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\") " pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 04:18:30.131057 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:30.131040    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v9hc\" (UniqueName: \"kubernetes.io/projected/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-kube-api-access-7v9hc\") pod \"premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9\" (UID: \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\") " pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 04:18:30.131311 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:30.131107    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-model-cache\") pod \"premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9\" (UID: \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\") " pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 04:18:30.131618 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:30.131588    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-model-cache\") pod \"premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9\" (UID: \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\") " pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 04:18:30.132571 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:30.132544    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-tmp-dir\") pod \"premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9\" (UID: \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\") " pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 04:18:30.133470 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:30.133442    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-home\") pod \"premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9\" (UID: \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\") " pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 04:18:30.137755 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:30.137732    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-dshm\") pod \"premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9\" (UID: \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\") " pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 04:18:30.137955 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:30.137934    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-tls-certs\") pod \"premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9\" (UID: \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\") " pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 04:18:30.156646 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:30.156595    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v9hc\" (UniqueName: \"kubernetes.io/projected/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-kube-api-access-7v9hc\") pod \"premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9\" (UID: \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\") " pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 04:18:30.217250 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:30.217215    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 04:18:30.519942 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:30.519916    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"]
Oct 30 04:18:30.522506 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:18:30.522475    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod734ec4d9_3542_4a3b_ba5b_d47f3ddf048e.slice/crio-4645706347eec0c6b72aec939364d0026be939ca3d1f6ffd901caef1e73f502f WatchSource:0}: Error finding container 4645706347eec0c6b72aec939364d0026be939ca3d1f6ffd901caef1e73f502f: Status 404 returned error can't find the container with id 4645706347eec0c6b72aec939364d0026be939ca3d1f6ffd901caef1e73f502f
Oct 30 04:18:31.086560 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:31.086519    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9" event={"ID":"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e","Type":"ContainerStarted","Data":"5b02fe924e4d1b2bc384abbca1922484fd3c8c496723ad598292d2503880cfbd"}
Oct 30 04:18:31.087021 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:31.086573    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9" event={"ID":"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e","Type":"ContainerStarted","Data":"4645706347eec0c6b72aec939364d0026be939ca3d1f6ffd901caef1e73f502f"}
Oct 30 04:18:31.127631 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:31.127585    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9" podStartSLOduration=2.1275696 podStartE2EDuration="2.1275696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:31.12598038 +0000 UTC m=+1230.538375965" watchObservedRunningTime="2026-10-30 04:18:31.1275696 +0000 UTC m=+1230.539965173"
Oct 30 04:18:31.217473 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:31.217446    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 04:18:33.731258 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:33.731212    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:18:40.218044 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:40.218011    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 04:18:40.230305 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:40.230283    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 04:18:41.194410 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:41.194381    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 04:18:49.697390 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:49.697349    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"]
Oct 30 04:18:49.702791 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:49.702768    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 04:18:49.720230 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:49.720201    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"]
Oct 30 04:18:49.729109 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:49.729080    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-6796b1\"/\"openshift-service-ca.crt\""
Oct 30 04:18:49.729109 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:49.729083    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-6796b1\"/\"kube-root-ca.crt\""
Oct 30 04:18:49.729302 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:49.729162    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6796b1\"/\"premium-w1-6796b1-kserve-self-signed-certs\""
Oct 30 04:18:49.729302 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:49.729180    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6796b1\"/\"default-dockercfg-zcchk\""
Oct 30 04:18:49.782351 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:49.782313    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-tmp-dir\") pod \"premium-w1-6796b1-kserve-578b7d867d-glskx\" (UID: \"70ea964b-efdb-4d99-b4f5-e3cde4331254\") " pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 04:18:49.782351 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:49.782350    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ea964b-efdb-4d99-b4f5-e3cde4331254-tls-certs\") pod \"premium-w1-6796b1-kserve-578b7d867d-glskx\" (UID: \"70ea964b-efdb-4d99-b4f5-e3cde4331254\") " pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 04:18:49.782571 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:49.782405    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-dshm\") pod \"premium-w1-6796b1-kserve-578b7d867d-glskx\" (UID: \"70ea964b-efdb-4d99-b4f5-e3cde4331254\") " pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 04:18:49.782571 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:49.782508    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfltx\" (UniqueName: \"kubernetes.io/projected/70ea964b-efdb-4d99-b4f5-e3cde4331254-kube-api-access-pfltx\") pod \"premium-w1-6796b1-kserve-578b7d867d-glskx\" (UID: \"70ea964b-efdb-4d99-b4f5-e3cde4331254\") " pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 04:18:49.782571 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:49.782531    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-home\") pod \"premium-w1-6796b1-kserve-578b7d867d-glskx\" (UID: \"70ea964b-efdb-4d99-b4f5-e3cde4331254\") " pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 04:18:49.782787 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:49.782762    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-model-cache\") pod \"premium-w1-6796b1-kserve-578b7d867d-glskx\" (UID: \"70ea964b-efdb-4d99-b4f5-e3cde4331254\") " pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 04:18:49.902687 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:49.902648    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-tmp-dir\") pod \"premium-w1-6796b1-kserve-578b7d867d-glskx\" (UID: \"70ea964b-efdb-4d99-b4f5-e3cde4331254\") " pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 04:18:49.902953 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:49.902921    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ea964b-efdb-4d99-b4f5-e3cde4331254-tls-certs\") pod \"premium-w1-6796b1-kserve-578b7d867d-glskx\" (UID: \"70ea964b-efdb-4d99-b4f5-e3cde4331254\") " pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 04:18:49.903055 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:49.902968    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-dshm\") pod \"premium-w1-6796b1-kserve-578b7d867d-glskx\" (UID: \"70ea964b-efdb-4d99-b4f5-e3cde4331254\") " pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 04:18:49.903055 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:49.903026    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfltx\" (UniqueName: \"kubernetes.io/projected/70ea964b-efdb-4d99-b4f5-e3cde4331254-kube-api-access-pfltx\") pod \"premium-w1-6796b1-kserve-578b7d867d-glskx\" (UID: \"70ea964b-efdb-4d99-b4f5-e3cde4331254\") " pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 04:18:49.903170 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:49.903058    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-home\") pod \"premium-w1-6796b1-kserve-578b7d867d-glskx\" (UID: \"70ea964b-efdb-4d99-b4f5-e3cde4331254\") " pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 04:18:49.903170 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:49.903064    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-tmp-dir\") pod \"premium-w1-6796b1-kserve-578b7d867d-glskx\" (UID: \"70ea964b-efdb-4d99-b4f5-e3cde4331254\") " pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 04:18:49.903170 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:49.903126    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-model-cache\") pod \"premium-w1-6796b1-kserve-578b7d867d-glskx\" (UID: \"70ea964b-efdb-4d99-b4f5-e3cde4331254\") " pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 04:18:49.903422 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:49.903404    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-home\") pod \"premium-w1-6796b1-kserve-578b7d867d-glskx\" (UID: \"70ea964b-efdb-4d99-b4f5-e3cde4331254\") " pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 04:18:49.903515 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:49.903494    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-model-cache\") pod \"premium-w1-6796b1-kserve-578b7d867d-glskx\" (UID: \"70ea964b-efdb-4d99-b4f5-e3cde4331254\") " pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 04:18:49.906891 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:49.906868    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ea964b-efdb-4d99-b4f5-e3cde4331254-tls-certs\") pod \"premium-w1-6796b1-kserve-578b7d867d-glskx\" (UID: \"70ea964b-efdb-4d99-b4f5-e3cde4331254\") " pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 04:18:49.912662 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:49.912639    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-dshm\") pod \"premium-w1-6796b1-kserve-578b7d867d-glskx\" (UID: \"70ea964b-efdb-4d99-b4f5-e3cde4331254\") " pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 04:18:49.932086 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:49.932044    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfltx\" (UniqueName: \"kubernetes.io/projected/70ea964b-efdb-4d99-b4f5-e3cde4331254-kube-api-access-pfltx\") pod \"premium-w1-6796b1-kserve-578b7d867d-glskx\" (UID: \"70ea964b-efdb-4d99-b4f5-e3cde4331254\") " pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 04:18:50.017000 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:50.016971    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 04:18:50.068308 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:50.068267    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:18:50.322349 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:50.322274    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"]
Oct 30 04:18:50.326149 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:18:50.326119    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70ea964b_efdb_4d99_b4f5_e3cde4331254.slice/crio-01a9abdd1d1c671018271e21649a5ee6c58f3f04348fa6f893ca3cb745c8e4f6 WatchSource:0}: Error finding container 01a9abdd1d1c671018271e21649a5ee6c58f3f04348fa6f893ca3cb745c8e4f6: Status 404 returned error can't find the container with id 01a9abdd1d1c671018271e21649a5ee6c58f3f04348fa6f893ca3cb745c8e4f6
Oct 30 04:18:51.240583 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:51.240540    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx" event={"ID":"70ea964b-efdb-4d99-b4f5-e3cde4331254","Type":"ContainerStarted","Data":"475b08288960784cda9d585c8d84e3ed5fb19d7f6a246f17942db07e87850e06"}
Oct 30 04:18:51.241063 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:51.240599    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx" event={"ID":"70ea964b-efdb-4d99-b4f5-e3cde4331254","Type":"ContainerStarted","Data":"01a9abdd1d1c671018271e21649a5ee6c58f3f04348fa6f893ca3cb745c8e4f6"}
Oct 30 04:18:51.302263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:51.298305    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx" podStartSLOduration=2.2982869790000002 podStartE2EDuration="2.298286979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:51.290373548 +0000 UTC m=+1250.702769135" watchObservedRunningTime="2026-10-30 04:18:51.298286979 +0000 UTC m=+1250.710682588"
Oct 30 04:18:52.017748 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:18:52.017718    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 04:18:55.608369 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:18:55.608334    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:19:00.017730 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:00.017696    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 04:19:00.030462 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:00.030434    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 04:19:00.303615 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:00.303539    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 04:19:12.470387 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:12.470346    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:19:23.102046 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.102014    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"]
Oct 30 04:19:23.105743 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.105724    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.131643 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.131610    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-6796b1-openshift-default-dockercfg-h254p\""
Oct 30 04:19:23.142274 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.142222    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"]
Oct 30 04:19:23.180550 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.180523    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c71f183b-8587-4f4f-a31f-203cb70db474-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.180672 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.180573    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.180672 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.180648    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.180743 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.180678    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfstt\" (UniqueName: \"kubernetes.io/projected/c71f183b-8587-4f4f-a31f-203cb70db474-kube-api-access-kfstt\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.180824 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.180761    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.180858 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.180828    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.180891 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.180858    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c71f183b-8587-4f4f-a31f-203cb70db474-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.180927 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.180893    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c71f183b-8587-4f4f-a31f-203cb70db474-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.180978 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.180964    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.281540 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.281509    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c71f183b-8587-4f4f-a31f-203cb70db474-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.281671 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.281567    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.281671 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.281595    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.281671 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.281620    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfstt\" (UniqueName: \"kubernetes.io/projected/c71f183b-8587-4f4f-a31f-203cb70db474-kube-api-access-kfstt\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.281671 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.281657    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.281876 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.281691    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.281876 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.281733    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c71f183b-8587-4f4f-a31f-203cb70db474-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.281876 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.281775    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c71f183b-8587-4f4f-a31f-203cb70db474-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.281876 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.281815    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.282146 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.282121    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.282226 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.282183    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.282455 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.282433    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.282455 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.282449    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.282680 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.282659    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c71f183b-8587-4f4f-a31f-203cb70db474-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.283817 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.283794    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.284116 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.284099    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c71f183b-8587-4f4f-a31f-203cb70db474-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.312058 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.312033    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c71f183b-8587-4f4f-a31f-203cb70db474-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.312192 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.312174    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfstt\" (UniqueName: \"kubernetes.io/projected/c71f183b-8587-4f4f-a31f-203cb70db474-kube-api-access-kfstt\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.418682 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.418610    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:23.738562 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.738535    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"]
Oct 30 04:19:23.741328 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:19:23.741300    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71f183b_8587_4f4f_a31f_203cb70db474.slice/crio-96f3bf0f5611bcfc1c5ed794db2d52a08f63167ab7b0fa843e982fbe6e09144a WatchSource:0}: Error finding container 96f3bf0f5611bcfc1c5ed794db2d52a08f63167ab7b0fa843e982fbe6e09144a: Status 404 returned error can't find the container with id 96f3bf0f5611bcfc1c5ed794db2d52a08f63167ab7b0fa843e982fbe6e09144a
Oct 30 04:19:23.743610 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.743577    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:19:23.743723 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.743646    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:19:23.743723 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:23.743678    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:19:24.417485 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:24.417445    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5" event={"ID":"c71f183b-8587-4f4f-a31f-203cb70db474","Type":"ContainerStarted","Data":"dfeaa2e75c1e0c5a7fa0079d5efe130ac969b8bd88f0a684b124967d7023a406"}
Oct 30 04:19:24.417870 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:24.417531    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5" event={"ID":"c71f183b-8587-4f4f-a31f-203cb70db474","Type":"ContainerStarted","Data":"96f3bf0f5611bcfc1c5ed794db2d52a08f63167ab7b0fa843e982fbe6e09144a"}
Oct 30 04:19:24.418869 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:24.418846    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:24.420426 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:24.420403    4874 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 04:19:24.420527 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:24.420449    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5" podUID="c71f183b-8587-4f4f-a31f-203cb70db474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 04:19:24.496480 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:24.496410    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5" podStartSLOduration=1.49638458 podStartE2EDuration="1.49638458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:24.495664179 +0000 UTC m=+1283.908059788" watchObservedRunningTime="2026-10-30 04:19:24.49638458 +0000 UTC m=+1283.908780155"
Oct 30 04:19:25.418991 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:25.418960    4874 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 04:19:25.419419 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:25.419011    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5" podUID="c71f183b-8587-4f4f-a31f-203cb70db474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 04:19:25.615093 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:19:25.615058    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:19:26.419434 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:26.419398    4874 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 04:19:26.419847 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:26.419466    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5" podUID="c71f183b-8587-4f4f-a31f-203cb70db474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 04:19:27.419074 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:27.419041    4874 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 04:19:27.419260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:27.419103    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5" podUID="c71f183b-8587-4f4f-a31f-203cb70db474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 04:19:28.419006 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:28.418970    4874 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 04:19:28.419412 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:28.419031    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5" podUID="c71f183b-8587-4f4f-a31f-203cb70db474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 04:19:29.419023 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:29.418981    4874 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 04:19:29.419522 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:29.419041    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5" podUID="c71f183b-8587-4f4f-a31f-203cb70db474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 04:19:30.436691 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:30.436654    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:30.437181 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:30.437048    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:30.438833 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:30.438808    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:19:35.677512 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.677470    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"]
Oct 30 04:19:35.681828 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.681804    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.711712 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.711682    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"]
Oct 30 04:19:35.731254 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.731206    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.731527 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.731504    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a80cae8-0b58-4643-aadb-771c6ea2651a-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.731730 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.731710    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.731924 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.731906    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.732077 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.732060    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj8p9\" (UniqueName: \"kubernetes.io/projected/6a80cae8-0b58-4643-aadb-771c6ea2651a-kube-api-access-jj8p9\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.732306 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.732288    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.732499 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.732483    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.732669 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.732654    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.732813 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.732797    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.834058 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.834027    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.834218 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.834070    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.834218 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.834092    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.834218 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.834133    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.834218 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.834156    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a80cae8-0b58-4643-aadb-771c6ea2651a-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.834218 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.834200    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.834488 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.834282    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.834614 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.834587    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj8p9\" (UniqueName: \"kubernetes.io/projected/6a80cae8-0b58-4643-aadb-771c6ea2651a-kube-api-access-jj8p9\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.834678 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.834611    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.834678 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.834642    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.834798 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.834730    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.834798 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.834782    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.835029 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.834998    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a80cae8-0b58-4643-aadb-771c6ea2651a-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.835106 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.835048    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.836585 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.836564    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.836698 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.836679    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.856615 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.856594    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.856705 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.856666    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj8p9\" (UniqueName: \"kubernetes.io/projected/6a80cae8-0b58-4643-aadb-771c6ea2651a-kube-api-access-jj8p9\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:35.997547 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:35.997519    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:36.324691 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:36.324664    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"]
Oct 30 04:19:36.327399 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:19:36.327369    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a80cae8_0b58_4643_aadb_771c6ea2651a.slice/crio-037cc8165f0bcd62a00d27c073fba4086d399a0290aa4f3ad867d986862dc05a WatchSource:0}: Error finding container 037cc8165f0bcd62a00d27c073fba4086d399a0290aa4f3ad867d986862dc05a: Status 404 returned error can't find the container with id 037cc8165f0bcd62a00d27c073fba4086d399a0290aa4f3ad867d986862dc05a
Oct 30 04:19:36.329646 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:36.329611    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:19:36.329751 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:36.329695    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:19:36.329751 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:36.329737    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:19:36.485175 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:36.485142    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh" event={"ID":"6a80cae8-0b58-4643-aadb-771c6ea2651a","Type":"ContainerStarted","Data":"85184c52773421c64459159dfaf6c0029b07dce151dca6d02948879f987788f4"}
Oct 30 04:19:36.485393 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:36.485195    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh" event={"ID":"6a80cae8-0b58-4643-aadb-771c6ea2651a","Type":"ContainerStarted","Data":"037cc8165f0bcd62a00d27c073fba4086d399a0290aa4f3ad867d986862dc05a"}
Oct 30 04:19:36.532702 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:36.532640    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh" podStartSLOduration=1.53262212 podStartE2EDuration="1.53262212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:36.529571128 +0000 UTC m=+1295.941966701" watchObservedRunningTime="2026-10-30 04:19:36.53262212 +0000 UTC m=+1295.945017694"
Oct 30 04:19:36.998158 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:36.998112    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:37.999389 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:37.999356    4874 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 04:19:37.999766 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:37.999417    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh" podUID="6a80cae8-0b58-4643-aadb-771c6ea2651a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 04:19:38.998370 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:38.998334    4874 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 04:19:38.998538 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:38.998400    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh" podUID="6a80cae8-0b58-4643-aadb-771c6ea2651a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 04:19:39.998262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:39.998011    4874 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 04:19:39.998262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:39.998085    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh" podUID="6a80cae8-0b58-4643-aadb-771c6ea2651a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 04:19:40.998378 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:40.998340    4874 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 04:19:40.998900 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:40.998435    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh" podUID="6a80cae8-0b58-4643-aadb-771c6ea2651a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 04:19:41.998798 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:41.998763    4874 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 04:19:41.999191 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:41.998812    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh" podUID="6a80cae8-0b58-4643-aadb-771c6ea2651a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 04:19:43.001998 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:43.001969    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:43.002429 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:43.002266    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:43.002994 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:43.002974    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:19:48.964724 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:48.964681    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:19:53.241807 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.241771    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"]
Oct 30 04:19:53.245483 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.245461    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.259344 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.259324    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-a2f984-openshift-default-dockercfg-pqhlj\""
Oct 30 04:19:53.273380 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.273357    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"]
Oct 30 04:19:53.367387 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.367351    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-workload-certs\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.367561 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.367394    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-workload-socket\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.367561 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.367531    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l896t\" (UniqueName: \"kubernetes.io/projected/16811f80-e367-4e31-b03c-82efdff4a3bb-kube-api-access-l896t\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.367685 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.367634    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-envoy\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.367685 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.367673    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-podinfo\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.367776 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.367763    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-credential-socket\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.367824 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.367796    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16811f80-e367-4e31-b03c-82efdff4a3bb-istiod-ca-cert\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.367898 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.367880    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-token\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.367958 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.367932    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-data\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.468718 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.468687    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-workload-socket\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.468888 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.468742    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l896t\" (UniqueName: \"kubernetes.io/projected/16811f80-e367-4e31-b03c-82efdff4a3bb-kube-api-access-l896t\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.468888 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.468798    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-envoy\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.468888 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.468829    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-podinfo\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.468888 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.468881    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-credential-socket\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.469102 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.468923    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16811f80-e367-4e31-b03c-82efdff4a3bb-istiod-ca-cert\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.469102 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.468967    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-token\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.469102 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.469002    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-data\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.469102 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.469058    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-workload-certs\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.469303 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.469131    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-workload-socket\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.469345 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.469318    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-credential-socket\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.469412 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.469368    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-data\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.469465 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.469436    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-workload-certs\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.469804 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.469776    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16811f80-e367-4e31-b03c-82efdff4a3bb-istiod-ca-cert\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.471132 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.471110    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-envoy\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.471377 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.471358    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-podinfo\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.491037 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.491011    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l896t\" (UniqueName: \"kubernetes.io/projected/16811f80-e367-4e31-b03c-82efdff4a3bb-kube-api-access-l896t\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.491129 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.491077    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-token\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.559457 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.559390    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:53.878291 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.878192    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"]
Oct 30 04:19:53.880482 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:19:53.880454    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16811f80_e367_4e31_b03c_82efdff4a3bb.slice/crio-a4eadbe208737834046f7d6187c9200ab7b71a24234f5828024c3de62ac3d99c WatchSource:0}: Error finding container a4eadbe208737834046f7d6187c9200ab7b71a24234f5828024c3de62ac3d99c: Status 404 returned error can't find the container with id a4eadbe208737834046f7d6187c9200ab7b71a24234f5828024c3de62ac3d99c
Oct 30 04:19:53.882782 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.882751    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:19:53.882869 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.882811    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:19:53.882869 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:53.882840    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:19:54.600485 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:54.600447    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65" event={"ID":"16811f80-e367-4e31-b03c-82efdff4a3bb","Type":"ContainerStarted","Data":"6ff68326899b4d992d4da129b13d399c544a3a7c4927637743b138660800de49"}
Oct 30 04:19:54.600892 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:54.600546    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65" event={"ID":"16811f80-e367-4e31-b03c-82efdff4a3bb","Type":"ContainerStarted","Data":"a4eadbe208737834046f7d6187c9200ab7b71a24234f5828024c3de62ac3d99c"}
Oct 30 04:19:54.673264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:54.671975    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65" podStartSLOduration=1.671956465 podStartE2EDuration="1.671956465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:54.654674511 +0000 UTC m=+1314.067070085" watchObservedRunningTime="2026-10-30 04:19:54.671956465 +0000 UTC m=+1314.084352039"
Oct 30 04:19:55.560309 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:55.560274    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:19:55.561413 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:55.561392    4874 patch_prober.go:29] interesting pod/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 04:19:55.561516 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:55.561454    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65" podUID="16811f80-e367-4e31-b03c-82efdff4a3bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 04:19:55.620197 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:19:55.620159    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:19:56.560814 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:56.560778    4874 patch_prober.go:29] interesting pod/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 04:19:56.560994 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:56.560834    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65" podUID="16811f80-e367-4e31-b03c-82efdff4a3bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 04:19:57.560012 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:57.559973    4874 patch_prober.go:29] interesting pod/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 04:19:57.560440 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:57.560034    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65" podUID="16811f80-e367-4e31-b03c-82efdff4a3bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 04:19:58.290098 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.290062    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"]
Oct 30 04:19:58.294872 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.294847    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 04:19:58.314715 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.314688    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"]
Oct 30 04:19:58.321609 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.321578    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-a2f984\"/\"kube-root-ca.crt\""
Oct 30 04:19:58.321784 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.321719    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a2f984\"/\"distinct-w4-a2f984-kserve-self-signed-certs\""
Oct 30 04:19:58.321784 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.321725    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a2f984\"/\"default-dockercfg-sqvb6\""
Oct 30 04:19:58.321784 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.321752    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-a2f984\"/\"openshift-service-ca.crt\""
Oct 30 04:19:58.349370 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.349340    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47a05334-0a58-4dd9-a9e3-2194031456cd-tls-certs\") pod \"distinct-w4-a2f984-kserve-54b7989d4c-74bg8\" (UID: \"47a05334-0a58-4dd9-a9e3-2194031456cd\") " pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 04:19:58.349487 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.349395    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-home\") pod \"distinct-w4-a2f984-kserve-54b7989d4c-74bg8\" (UID: \"47a05334-0a58-4dd9-a9e3-2194031456cd\") " pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 04:19:58.349487 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.349458    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-tmp-dir\") pod \"distinct-w4-a2f984-kserve-54b7989d4c-74bg8\" (UID: \"47a05334-0a58-4dd9-a9e3-2194031456cd\") " pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 04:19:58.349575 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.349530    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpxvr\" (UniqueName: \"kubernetes.io/projected/47a05334-0a58-4dd9-a9e3-2194031456cd-kube-api-access-qpxvr\") pod \"distinct-w4-a2f984-kserve-54b7989d4c-74bg8\" (UID: \"47a05334-0a58-4dd9-a9e3-2194031456cd\") " pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 04:19:58.349575 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.349556    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-dshm\") pod \"distinct-w4-a2f984-kserve-54b7989d4c-74bg8\" (UID: \"47a05334-0a58-4dd9-a9e3-2194031456cd\") " pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 04:19:58.349674 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.349632    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-model-cache\") pod \"distinct-w4-a2f984-kserve-54b7989d4c-74bg8\" (UID: \"47a05334-0a58-4dd9-a9e3-2194031456cd\") " pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 04:19:58.450729 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.450696    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-home\") pod \"distinct-w4-a2f984-kserve-54b7989d4c-74bg8\" (UID: \"47a05334-0a58-4dd9-a9e3-2194031456cd\") " pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 04:19:58.450729 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.450726    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-tmp-dir\") pod \"distinct-w4-a2f984-kserve-54b7989d4c-74bg8\" (UID: \"47a05334-0a58-4dd9-a9e3-2194031456cd\") " pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 04:19:58.450950 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.450753    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpxvr\" (UniqueName: \"kubernetes.io/projected/47a05334-0a58-4dd9-a9e3-2194031456cd-kube-api-access-qpxvr\") pod \"distinct-w4-a2f984-kserve-54b7989d4c-74bg8\" (UID: \"47a05334-0a58-4dd9-a9e3-2194031456cd\") " pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 04:19:58.450950 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.450771    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-dshm\") pod \"distinct-w4-a2f984-kserve-54b7989d4c-74bg8\" (UID: \"47a05334-0a58-4dd9-a9e3-2194031456cd\") " pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 04:19:58.450950 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.450814    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-model-cache\") pod \"distinct-w4-a2f984-kserve-54b7989d4c-74bg8\" (UID: \"47a05334-0a58-4dd9-a9e3-2194031456cd\") " pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 04:19:58.450950 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.450938    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47a05334-0a58-4dd9-a9e3-2194031456cd-tls-certs\") pod \"distinct-w4-a2f984-kserve-54b7989d4c-74bg8\" (UID: \"47a05334-0a58-4dd9-a9e3-2194031456cd\") " pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 04:19:58.451176 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.451145    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-tmp-dir\") pod \"distinct-w4-a2f984-kserve-54b7989d4c-74bg8\" (UID: \"47a05334-0a58-4dd9-a9e3-2194031456cd\") " pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 04:19:58.451176 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.451159    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-home\") pod \"distinct-w4-a2f984-kserve-54b7989d4c-74bg8\" (UID: \"47a05334-0a58-4dd9-a9e3-2194031456cd\") " pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 04:19:58.451408 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.451267    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-model-cache\") pod \"distinct-w4-a2f984-kserve-54b7989d4c-74bg8\" (UID: \"47a05334-0a58-4dd9-a9e3-2194031456cd\") " pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 04:19:58.453616 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.453592    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-dshm\") pod \"distinct-w4-a2f984-kserve-54b7989d4c-74bg8\" (UID: \"47a05334-0a58-4dd9-a9e3-2194031456cd\") " pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 04:19:58.453777 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.453759    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47a05334-0a58-4dd9-a9e3-2194031456cd-tls-certs\") pod \"distinct-w4-a2f984-kserve-54b7989d4c-74bg8\" (UID: \"47a05334-0a58-4dd9-a9e3-2194031456cd\") " pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 04:19:58.490261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.486638    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpxvr\" (UniqueName: \"kubernetes.io/projected/47a05334-0a58-4dd9-a9e3-2194031456cd-kube-api-access-qpxvr\") pod \"distinct-w4-a2f984-kserve-54b7989d4c-74bg8\" (UID: \"47a05334-0a58-4dd9-a9e3-2194031456cd\") " pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 04:19:58.559900 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.559798    4874 patch_prober.go:29] interesting pod/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 04:19:58.560065 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.559891    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65" podUID="16811f80-e367-4e31-b03c-82efdff4a3bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 04:19:58.608639 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.608604    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 04:19:58.917655 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:58.917630    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"]
Oct 30 04:19:58.920271 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:19:58.920225    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47a05334_0a58_4dd9_a9e3_2194031456cd.slice/crio-7cae1443273757d55a1ffceba833d72189204819b2c41bff8b02717a3f790e65 WatchSource:0}: Error finding container 7cae1443273757d55a1ffceba833d72189204819b2c41bff8b02717a3f790e65: Status 404 returned error can't find the container with id 7cae1443273757d55a1ffceba833d72189204819b2c41bff8b02717a3f790e65
Oct 30 04:19:59.560568 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:59.560531    4874 patch_prober.go:29] interesting pod/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 04:19:59.560994 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:59.560586    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65" podUID="16811f80-e367-4e31-b03c-82efdff4a3bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 04:19:59.633442 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:59.633396    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8" event={"ID":"47a05334-0a58-4dd9-a9e3-2194031456cd","Type":"ContainerStarted","Data":"88ea629527c40f3ba8f6bf9aa6c2687a41b1dd82e253e759f61c3011aec823cf"}
Oct 30 04:19:59.633442 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:59.633445    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8" event={"ID":"47a05334-0a58-4dd9-a9e3-2194031456cd","Type":"ContainerStarted","Data":"7cae1443273757d55a1ffceba833d72189204819b2c41bff8b02717a3f790e65"}
Oct 30 04:19:59.679265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:19:59.675009    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8" podStartSLOduration=1.6749905859999998 podStartE2EDuration="1.674990586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:59.674591279 +0000 UTC m=+1319.086986853" watchObservedRunningTime="2026-10-30 04:19:59.674990586 +0000 UTC m=+1319.087386160"
Oct 30 04:20:00.609553 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:00.609510    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 04:20:00.736601 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:00.736570    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:20:00.737184 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:00.737141    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:20:00.738602 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:00.738579    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:20:03.053260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:03.052937    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:20:08.609029 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:08.608994    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 04:20:08.621396 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:08.621373    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 04:20:08.728394 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:08.728364    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 04:20:10.733089 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:10.733059    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"]
Oct 30 04:20:10.737877 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:10.737854    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:10.759801 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:10.759772    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"]
Oct 30 04:20:10.767511 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:10.767488    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6796b1\"/\"e2e-conflict-b-54a3def2-kserve-self-signed-certs\""
Oct 30 04:20:10.805111 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:10.805082    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-model-cache\") pod \"e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj\" (UID: \"070fef5e-43a2-4560-baa6-e90359388370\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:10.805249 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:10.805117    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-home\") pod \"e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj\" (UID: \"070fef5e-43a2-4560-baa6-e90359388370\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:10.805451 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:10.805424    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-dshm\") pod \"e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj\" (UID: \"070fef5e-43a2-4560-baa6-e90359388370\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:10.805575 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:10.805472    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-tmp-dir\") pod \"e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj\" (UID: \"070fef5e-43a2-4560-baa6-e90359388370\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:10.805575 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:10.805497    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s65vk\" (UniqueName: \"kubernetes.io/projected/070fef5e-43a2-4560-baa6-e90359388370-kube-api-access-s65vk\") pod \"e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj\" (UID: \"070fef5e-43a2-4560-baa6-e90359388370\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:10.805575 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:10.805533    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/070fef5e-43a2-4560-baa6-e90359388370-tls-certs\") pod \"e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj\" (UID: \"070fef5e-43a2-4560-baa6-e90359388370\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:10.923265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:10.920739    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-model-cache\") pod \"e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj\" (UID: \"070fef5e-43a2-4560-baa6-e90359388370\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:10.923265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:10.920807    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-home\") pod \"e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj\" (UID: \"070fef5e-43a2-4560-baa6-e90359388370\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:10.923265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:10.920977    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-dshm\") pod \"e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj\" (UID: \"070fef5e-43a2-4560-baa6-e90359388370\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:10.923265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:10.921010    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-tmp-dir\") pod \"e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj\" (UID: \"070fef5e-43a2-4560-baa6-e90359388370\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:10.923265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:10.921041    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s65vk\" (UniqueName: \"kubernetes.io/projected/070fef5e-43a2-4560-baa6-e90359388370-kube-api-access-s65vk\") pod \"e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj\" (UID: \"070fef5e-43a2-4560-baa6-e90359388370\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:10.923265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:10.921093    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/070fef5e-43a2-4560-baa6-e90359388370-tls-certs\") pod \"e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj\" (UID: \"070fef5e-43a2-4560-baa6-e90359388370\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:10.924570 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:10.924537    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-tmp-dir\") pod \"e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj\" (UID: \"070fef5e-43a2-4560-baa6-e90359388370\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:10.929729 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:10.929697    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-home\") pod \"e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj\" (UID: \"070fef5e-43a2-4560-baa6-e90359388370\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:10.929990 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:10.929970    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-model-cache\") pod \"e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj\" (UID: \"070fef5e-43a2-4560-baa6-e90359388370\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:10.953088 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:10.953021    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-dshm\") pod \"e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj\" (UID: \"070fef5e-43a2-4560-baa6-e90359388370\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:10.956229 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:10.953816    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/070fef5e-43a2-4560-baa6-e90359388370-tls-certs\") pod \"e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj\" (UID: \"070fef5e-43a2-4560-baa6-e90359388370\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:10.961861 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:10.961812    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s65vk\" (UniqueName: \"kubernetes.io/projected/070fef5e-43a2-4560-baa6-e90359388370-kube-api-access-s65vk\") pod \"e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj\" (UID: \"070fef5e-43a2-4560-baa6-e90359388370\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:11.051768 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:11.051690    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:11.353960 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:11.353936    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"]
Oct 30 04:20:11.356543 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:20:11.356511    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod070fef5e_43a2_4560_baa6_e90359388370.slice/crio-9c283d15d18f8067dc6a290d81768473246d98d9297e5e108be36fcf1349afe4 WatchSource:0}: Error finding container 9c283d15d18f8067dc6a290d81768473246d98d9297e5e108be36fcf1349afe4: Status 404 returned error can't find the container with id 9c283d15d18f8067dc6a290d81768473246d98d9297e5e108be36fcf1349afe4
Oct 30 04:20:11.734613 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:11.734579    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj" event={"ID":"070fef5e-43a2-4560-baa6-e90359388370","Type":"ContainerStarted","Data":"c2c9d7c2d440c82ea14da326486a9b1811aa8b1f99dba79e955a394bc617bbfc"}
Oct 30 04:20:11.734979 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:11.734620    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj" event={"ID":"070fef5e-43a2-4560-baa6-e90359388370","Type":"ContainerStarted","Data":"9c283d15d18f8067dc6a290d81768473246d98d9297e5e108be36fcf1349afe4"}
Oct 30 04:20:11.774739 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:11.774679    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj" podStartSLOduration=1.774662363 podStartE2EDuration="1.774662363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:11.773478369 +0000 UTC m=+1331.185873942" watchObservedRunningTime="2026-10-30 04:20:11.774662363 +0000 UTC m=+1331.187057939"
Oct 30 04:20:12.052733 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:12.052655    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:21.052805 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:21.052770    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:21.065445 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:21.065423    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:21.797629 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:21.797596    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:25.609127 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:20:25.609093    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:20:27.355094 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:27.355063    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:20:33.801958 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:33.801916    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"]
Oct 30 04:20:38.327445 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.327408    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"]
Oct 30 04:20:38.332349 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.332330    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.362050 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.362017    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"]
Oct 30 04:20:38.485583 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.485548    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8247\" (UniqueName: \"kubernetes.io/projected/37fde430-0d9b-4c2e-b4d2-97e36fa58004-kube-api-access-v8247\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.485583 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.485587    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.485803 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.485608    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.485803 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.485776    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.485895 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.485819    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.485895 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.485839    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.485895 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.485875    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.486011 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.485926    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.486011 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.485957    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.586933 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.586838    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.586933 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.586877    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.586933 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.586907    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.587204 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.586939    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.587204 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.586980    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.587204 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.587066    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8247\" (UniqueName: \"kubernetes.io/projected/37fde430-0d9b-4c2e-b4d2-97e36fa58004-kube-api-access-v8247\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.587204 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.587097    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.587204 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.587136    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.587481 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.587286    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.587590 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.587565    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.587648 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.587592    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.587714 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.587691    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.587758 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.587723    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.588022 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.588004    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.589227 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.589203    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.589488 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.589469    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.609557 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.609535    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.609724 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.609705    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8247\" (UniqueName: \"kubernetes.io/projected/37fde430-0d9b-4c2e-b4d2-97e36fa58004-kube-api-access-v8247\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.646879 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.646853    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:38.963382 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.963357    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"]
Oct 30 04:20:38.965971 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:20:38.965945    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37fde430_0d9b_4c2e_b4d2_97e36fa58004.slice/crio-597ba0f3062b35b284562e4a6d75278c121c6b298a1e5a216d438fb0f20f2d32 WatchSource:0}: Error finding container 597ba0f3062b35b284562e4a6d75278c121c6b298a1e5a216d438fb0f20f2d32: Status 404 returned error can't find the container with id 597ba0f3062b35b284562e4a6d75278c121c6b298a1e5a216d438fb0f20f2d32
Oct 30 04:20:38.968310 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.968279    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:20:38.968389 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.968344    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:20:38.968389 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:38.968373    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:20:39.886974 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:39.886931    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6" event={"ID":"37fde430-0d9b-4c2e-b4d2-97e36fa58004","Type":"ContainerStarted","Data":"1a339d105362360e7fba03f9c60894b76b296fe77359cfe2cadbc15dbc6abe58"}
Oct 30 04:20:39.887745 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:39.887720    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6" event={"ID":"37fde430-0d9b-4c2e-b4d2-97e36fa58004","Type":"ContainerStarted","Data":"597ba0f3062b35b284562e4a6d75278c121c6b298a1e5a216d438fb0f20f2d32"}
Oct 30 04:20:39.944287 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:39.944214    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6" podStartSLOduration=1.944201627 podStartE2EDuration="1.944201627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:39.941940324 +0000 UTC m=+1359.354335896" watchObservedRunningTime="2026-10-30 04:20:39.944201627 +0000 UTC m=+1359.356597200"
Oct 30 04:20:40.647542 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:40.647506    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:40.649065 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:40.649042    4874 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 04:20:40.649152 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:40.649097    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6" podUID="37fde430-0d9b-4c2e-b4d2-97e36fa58004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 04:20:41.647454 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:41.647418    4874 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 04:20:41.647839 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:41.647478    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6" podUID="37fde430-0d9b-4c2e-b4d2-97e36fa58004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 04:20:42.647366 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:42.647332    4874 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 04:20:42.647551 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:42.647385    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6" podUID="37fde430-0d9b-4c2e-b4d2-97e36fa58004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 04:20:43.648042 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:43.648009    4874 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 04:20:43.648470 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:43.648069    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6" podUID="37fde430-0d9b-4c2e-b4d2-97e36fa58004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 04:20:44.647908 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:44.647866    4874 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 04:20:44.648091 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:44.647928    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6" podUID="37fde430-0d9b-4c2e-b4d2-97e36fa58004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 04:20:45.647733 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:45.647694    4874 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 04:20:45.647902 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:45.647759    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6" podUID="37fde430-0d9b-4c2e-b4d2-97e36fa58004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 04:20:47.102332 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:47.102299    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:47.102990 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:47.102965    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:47.104094 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:47.104069    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:20:48.822452 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:48.822403    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj" podUID="070fef5e-43a2-4560-baa6-e90359388370" containerName="main" containerID="cri-o://c2c9d7c2d440c82ea14da326486a9b1811aa8b1f99dba79e955a394bc617bbfc" gracePeriod=45
Oct 30 04:20:48.945662 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:48.945638    4874 generic.go:356] "Generic (PLEG): container finished" podID="070fef5e-43a2-4560-baa6-e90359388370" containerID="c2c9d7c2d440c82ea14da326486a9b1811aa8b1f99dba79e955a394bc617bbfc" exitCode=0
Oct 30 04:20:48.945793 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:48.945714    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj" event={"ID":"070fef5e-43a2-4560-baa6-e90359388370","Type":"ContainerDied","Data":"c2c9d7c2d440c82ea14da326486a9b1811aa8b1f99dba79e955a394bc617bbfc"}
Oct 30 04:20:49.128402 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:49.128377    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:49.215132 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:49.215107    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/070fef5e-43a2-4560-baa6-e90359388370-kube-api-access-s65vk\" (UniqueName: \"kubernetes.io/projected/070fef5e-43a2-4560-baa6-e90359388370-kube-api-access-s65vk\") pod \"070fef5e-43a2-4560-baa6-e90359388370\" (UID: \"070fef5e-43a2-4560-baa6-e90359388370\") "
Oct 30 04:20:49.215308 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:49.215184    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-home\" (UniqueName: \"kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-home\") pod \"070fef5e-43a2-4560-baa6-e90359388370\" (UID: \"070fef5e-43a2-4560-baa6-e90359388370\") "
Oct 30 04:20:49.215308 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:49.215211    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-model-cache\") pod \"070fef5e-43a2-4560-baa6-e90359388370\" (UID: \"070fef5e-43a2-4560-baa6-e90359388370\") "
Oct 30 04:20:49.215435 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:49.215309    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-dshm\" (UniqueName: \"kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-dshm\") pod \"070fef5e-43a2-4560-baa6-e90359388370\" (UID: \"070fef5e-43a2-4560-baa6-e90359388370\") "
Oct 30 04:20:49.215435 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:49.215354    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/070fef5e-43a2-4560-baa6-e90359388370-tls-certs\" (UniqueName: \"kubernetes.io/secret/070fef5e-43a2-4560-baa6-e90359388370-tls-certs\") pod \"070fef5e-43a2-4560-baa6-e90359388370\" (UID: \"070fef5e-43a2-4560-baa6-e90359388370\") "
Oct 30 04:20:49.215435 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:49.215410    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-tmp-dir\") pod \"070fef5e-43a2-4560-baa6-e90359388370\" (UID: \"070fef5e-43a2-4560-baa6-e90359388370\") "
Oct 30 04:20:49.215589 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:49.215488    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-home" pod "070fef5e-43a2-4560-baa6-e90359388370" (UID: "070fef5e-43a2-4560-baa6-e90359388370"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:20:49.215589 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:49.215523    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-model-cache" pod "070fef5e-43a2-4560-baa6-e90359388370" (UID: "070fef5e-43a2-4560-baa6-e90359388370"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:20:49.215835 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:49.215795    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-tmp-dir" pod "070fef5e-43a2-4560-baa6-e90359388370" (UID: "070fef5e-43a2-4560-baa6-e90359388370"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:20:49.215954 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:49.215835    4874 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-home\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:20:49.215954 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:49.215850    4874 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-model-cache\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:20:49.217567 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:49.217528    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/070fef5e-43a2-4560-baa6-e90359388370-tls-certs" pod "070fef5e-43a2-4560-baa6-e90359388370" (UID: "070fef5e-43a2-4560-baa6-e90359388370"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:20:49.217567 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:49.217534    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-dshm" pod "070fef5e-43a2-4560-baa6-e90359388370" (UID: "070fef5e-43a2-4560-baa6-e90359388370"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:20:49.217696 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:49.217580    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070fef5e-43a2-4560-baa6-e90359388370-kube-api-access-s65vk" pod "070fef5e-43a2-4560-baa6-e90359388370" (UID: "070fef5e-43a2-4560-baa6-e90359388370"). InnerVolumeSpecName "kube-api-access-s65vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:20:49.317029 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:49.317003    4874 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-tmp-dir\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:20:49.317029 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:49.317029    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s65vk\" (UniqueName: \"kubernetes.io/projected/070fef5e-43a2-4560-baa6-e90359388370-kube-api-access-s65vk\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:20:49.317192 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:49.317037    4874 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/070fef5e-43a2-4560-baa6-e90359388370-dshm\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:20:49.317192 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:49.317048    4874 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/070fef5e-43a2-4560-baa6-e90359388370-tls-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:20:49.951904 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:49.951874    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"
Oct 30 04:20:49.952370 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:49.951899    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj" event={"ID":"070fef5e-43a2-4560-baa6-e90359388370","Type":"ContainerDied","Data":"9c283d15d18f8067dc6a290d81768473246d98d9297e5e108be36fcf1349afe4"}
Oct 30 04:20:49.952370 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:49.951943    4874 scope.go:122] "RemoveContainer" containerID="c2c9d7c2d440c82ea14da326486a9b1811aa8b1f99dba79e955a394bc617bbfc"
Oct 30 04:20:50.001248 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:50.001214    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"]
Oct 30 04:20:50.012614 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:50.012592    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/e2e-conflict-b-54a3def2-kserve-7dc6874867-7mtsj"]
Oct 30 04:20:50.750849 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:50.750814    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="070fef5e-43a2-4560-baa6-e90359388370" path="/var/lib/kubelet/pods/070fef5e-43a2-4560-baa6-e90359388370/volumes"
Oct 30 04:20:51.328664 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.328632    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"]
Oct 30 04:20:51.329094 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.329080    4874 cpu_manager.go:414] "containerMap: removing container" podUID="070fef5e-43a2-4560-baa6-e90359388370" containerName="main"
Oct 30 04:20:51.329140 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.329094    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="070fef5e-43a2-4560-baa6-e90359388370" containerName="main"
Oct 30 04:20:51.329180 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.329171    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="070fef5e-43a2-4560-baa6-e90359388370" containerName="main"
Oct 30 04:20:51.332356 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.332336    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.355940 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.355904    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"]
Oct 30 04:20:51.445495 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.445463    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.445667 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.445510    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x54r\" (UniqueName: \"kubernetes.io/projected/29a55637-b48a-4cc9-8b2a-454dacf07855-kube-api-access-8x54r\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.445667 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.445608    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.445667 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.445652    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.445784 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.445745    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.445784 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.445777    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.445857 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.445835    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29a55637-b48a-4cc9-8b2a-454dacf07855-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.445911 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.445859    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.445952 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.445928    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.546511 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.546483    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.546681 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.546520    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.546681 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.546554    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29a55637-b48a-4cc9-8b2a-454dacf07855-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.546681 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.546587    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.546863 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.546778    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.546863 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.546819    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.546863 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.546840    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.547025 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.546865    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x54r\" (UniqueName: \"kubernetes.io/projected/29a55637-b48a-4cc9-8b2a-454dacf07855-kube-api-access-8x54r\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.547025 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.546913    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.547025 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.546939    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.547187 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.547022    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.547262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.547229    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.547330 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.547289    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.547383 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.547356    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29a55637-b48a-4cc9-8b2a-454dacf07855-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.548986 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.548964    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.549210 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.549191    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.569074 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.569043    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.569339 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.569318    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x54r\" (UniqueName: \"kubernetes.io/projected/29a55637-b48a-4cc9-8b2a-454dacf07855-kube-api-access-8x54r\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.646546 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.646478    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:51.734429 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.734399    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"]
Oct 30 04:20:51.738948 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.738924    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 04:20:51.762267 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.759823    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"]
Oct 30 04:20:51.762267 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.760004    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6796b1\"/\"distinct-w1-6796b1-kserve-self-signed-certs\""
Oct 30 04:20:51.849868 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.849841    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-tls-certs\") pod \"distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k\" (UID: \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\") " pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 04:20:51.850004 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.849876    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqjsc\" (UniqueName: \"kubernetes.io/projected/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-kube-api-access-mqjsc\") pod \"distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k\" (UID: \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\") " pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 04:20:51.850084 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.850009    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-home\") pod \"distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k\" (UID: \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\") " pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 04:20:51.850084 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.850041    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-model-cache\") pod \"distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k\" (UID: \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\") " pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 04:20:51.850202 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.850154    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-tmp-dir\") pod \"distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k\" (UID: \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\") " pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 04:20:51.850323 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.850302    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-dshm\") pod \"distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k\" (UID: \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\") " pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 04:20:51.951140 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.951069    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-tls-certs\") pod \"distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k\" (UID: \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\") " pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 04:20:51.951140 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.951105    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqjsc\" (UniqueName: \"kubernetes.io/projected/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-kube-api-access-mqjsc\") pod \"distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k\" (UID: \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\") " pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 04:20:51.951333 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.951294    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-home\") pod \"distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k\" (UID: \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\") " pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 04:20:51.951394 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.951339    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-model-cache\") pod \"distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k\" (UID: \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\") " pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 04:20:51.951447 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.951410    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-tmp-dir\") pod \"distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k\" (UID: \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\") " pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 04:20:51.951496 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.951468    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-dshm\") pod \"distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k\" (UID: \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\") " pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 04:20:51.951721 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.951699    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-home\") pod \"distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k\" (UID: \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\") " pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 04:20:51.951781 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.951722    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-model-cache\") pod \"distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k\" (UID: \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\") " pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 04:20:51.951819 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.951776    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-tmp-dir\") pod \"distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k\" (UID: \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\") " pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 04:20:51.953412 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.953391    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-dshm\") pod \"distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k\" (UID: \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\") " pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 04:20:51.953542 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.953525    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-tls-certs\") pod \"distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k\" (UID: \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\") " pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 04:20:51.993760 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:51.993305    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqjsc\" (UniqueName: \"kubernetes.io/projected/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-kube-api-access-mqjsc\") pod \"distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k\" (UID: \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\") " pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 04:20:52.061470 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:52.059190    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"]
Oct 30 04:20:52.061927 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:52.061900    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 04:20:52.064124 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:52.064079    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:20:52.064893 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:52.064856    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:20:52.065383 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:52.065303    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:20:52.385934 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:52.385899    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"]
Oct 30 04:20:52.389651 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:20:52.389614    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e28b1d5_cbaa_40ea_b74c_fecb28d84a38.slice/crio-05221d6414bf902a631e5bd81d7d60e4a9f7e44153a13f403fd30c140a69e20f WatchSource:0}: Error finding container 05221d6414bf902a631e5bd81d7d60e4a9f7e44153a13f403fd30c140a69e20f: Status 404 returned error can't find the container with id 05221d6414bf902a631e5bd81d7d60e4a9f7e44153a13f403fd30c140a69e20f
Oct 30 04:20:53.094591 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:53.094549    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k" event={"ID":"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38","Type":"ContainerStarted","Data":"73fad9de186bacea3765e08dc4af3d10a41eccb24fcffb8acd8a6baca0d87f15"}
Oct 30 04:20:53.094791 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:53.094602    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k" event={"ID":"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38","Type":"ContainerStarted","Data":"05221d6414bf902a631e5bd81d7d60e4a9f7e44153a13f403fd30c140a69e20f"}
Oct 30 04:20:53.096297 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:53.096270    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx" event={"ID":"29a55637-b48a-4cc9-8b2a-454dacf07855","Type":"ContainerStarted","Data":"2c9f3c78bf1a5ee618f745a38d046ac17b9f637ea5b5f3e8a019c6f6d12ef2b8"}
Oct 30 04:20:53.096409 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:53.096318    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx" event={"ID":"29a55637-b48a-4cc9-8b2a-454dacf07855","Type":"ContainerStarted","Data":"7279b66d39af6491cfa0285bdf38f739caa1354c30ecf59cb940c26e3bf78db6"}
Oct 30 04:20:53.136453 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:53.136413    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k" podStartSLOduration=2.136402732 podStartE2EDuration="2.136402732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:53.1344441 +0000 UTC m=+1372.546839673" watchObservedRunningTime="2026-10-30 04:20:53.136402732 +0000 UTC m=+1372.548798298"
Oct 30 04:20:53.647052 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:53.647017    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:53.648493 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:53.648469    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 04:20:53.648595 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:53.648513    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx" podUID="29a55637-b48a-4cc9-8b2a-454dacf07855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 04:20:54.089265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:54.080329    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 04:20:54.643869 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:54.643814    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx" podStartSLOduration=3.6437983149999997 podStartE2EDuration="3.643798315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:53.189636314 +0000 UTC m=+1372.602031888" watchObservedRunningTime="2026-10-30 04:20:54.643798315 +0000 UTC m=+1374.056193920"
Oct 30 04:20:54.645266 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:54.645231    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:20:54.647135 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:54.647112    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 04:20:54.647435 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:54.647158    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx" podUID="29a55637-b48a-4cc9-8b2a-454dacf07855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 04:20:55.613505 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:20:55.613469    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:20:55.647361 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:55.647334    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 04:20:55.647678 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:55.647378    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx" podUID="29a55637-b48a-4cc9-8b2a-454dacf07855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 04:20:56.647480 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:56.647447    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 04:20:56.647903 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:56.647509    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx" podUID="29a55637-b48a-4cc9-8b2a-454dacf07855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 04:20:57.647438 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:57.647392    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 04:20:57.647599 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:57.647463    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx" podUID="29a55637-b48a-4cc9-8b2a-454dacf07855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 04:20:58.647149 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:58.647114    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 04:20:58.647327 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:58.647173    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx" podUID="29a55637-b48a-4cc9-8b2a-454dacf07855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 04:20:59.651979 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:59.651948    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:59.652460 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:59.652418    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:20:59.653060 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:20:59.653038    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:21:02.069618 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:02.069583    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 04:21:02.082112 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:02.082082    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 04:21:02.154338 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:02.154307    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 04:21:09.921175 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:09.921146    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:21:23.462741 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.462648    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"]
Oct 30 04:21:23.466455 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.466435    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.497608 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.497576    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"]
Oct 30 04:21:23.576405 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.576368    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.576405 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.576404    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.576599 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.576428    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.576599 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.576570    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.576670 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.576621    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.576745 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.576726    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.576780 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.576752    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc017d56-de5c-4ce1-bb57-421f757c790e-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.576857 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.576841    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.576908 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.576896    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62n4x\" (UniqueName: \"kubernetes.io/projected/bc017d56-de5c-4ce1-bb57-421f757c790e-kube-api-access-62n4x\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.677970 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.677933    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62n4x\" (UniqueName: \"kubernetes.io/projected/bc017d56-de5c-4ce1-bb57-421f757c790e-kube-api-access-62n4x\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.678159 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.677999    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.678159 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.678021    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.678159 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.678051    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.678159 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.678090    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.678159 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.678138    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.678485 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.678206    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.678485 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.678265    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc017d56-de5c-4ce1-bb57-421f757c790e-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.678485 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.678326    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.678587 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.678568    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.678889 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.678863    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.678974 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.678951    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.679035 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.678983    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc017d56-de5c-4ce1-bb57-421f757c790e-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.679035 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.679015    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.680512 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.680495    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.680587 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.680560    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.702352 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.702322    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.702719 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.702699    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62n4x\" (UniqueName: \"kubernetes.io/projected/bc017d56-de5c-4ce1-bb57-421f757c790e-kube-api-access-62n4x\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:23.779649 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:23.779617    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:24.098452 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:24.098297    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"]
Oct 30 04:21:24.101250 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:21:24.101215    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc017d56_de5c_4ce1_bb57_421f757c790e.slice/crio-4379e0b3d15dceb169af4249e2c26510109bfe336cdbc7eecd3e50bf4f958b61 WatchSource:0}: Error finding container 4379e0b3d15dceb169af4249e2c26510109bfe336cdbc7eecd3e50bf4f958b61: Status 404 returned error can't find the container with id 4379e0b3d15dceb169af4249e2c26510109bfe336cdbc7eecd3e50bf4f958b61
Oct 30 04:21:24.103486 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:24.103455    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:21:24.103586 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:24.103530    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:21:24.103586 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:24.103572    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:21:24.306151 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:24.306118    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4" event={"ID":"bc017d56-de5c-4ce1-bb57-421f757c790e","Type":"ContainerStarted","Data":"90c85095f32956eb5d92da84fa92178e1075044114cdf456d75747075bed6547"}
Oct 30 04:21:24.306351 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:24.306170    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4" event={"ID":"bc017d56-de5c-4ce1-bb57-421f757c790e","Type":"ContainerStarted","Data":"4379e0b3d15dceb169af4249e2c26510109bfe336cdbc7eecd3e50bf4f958b61"}
Oct 30 04:21:24.352447 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:24.352349    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4" podStartSLOduration=1.352331415 podStartE2EDuration="1.352331415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:21:24.349984165 +0000 UTC m=+1403.762379737" watchObservedRunningTime="2026-10-30 04:21:24.352331415 +0000 UTC m=+1403.764726989"
Oct 30 04:21:24.785262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:24.780308    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:25.613627 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:21:25.613572    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:21:25.789264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:25.788797    4874 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 04:21:25.789264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:25.788855    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4" podUID="bc017d56-de5c-4ce1-bb57-421f757c790e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 04:21:26.789265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:26.783528    4874 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 04:21:26.789265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:26.783593    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4" podUID="bc017d56-de5c-4ce1-bb57-421f757c790e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 04:21:27.785036 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:27.784595    4874 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 04:21:27.785036 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:27.784665    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4" podUID="bc017d56-de5c-4ce1-bb57-421f757c790e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 04:21:28.780871 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:28.780821    4874 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 04:21:28.781507 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:28.781460    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4" podUID="bc017d56-de5c-4ce1-bb57-421f757c790e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 04:21:29.789635 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:29.789599    4874 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 04:21:29.790125 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:29.789734    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4" podUID="bc017d56-de5c-4ce1-bb57-421f757c790e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 04:21:30.953478 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:30.953440    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:30.954370 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:30.954345    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:30.955339 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:30.955315    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:21:49.411568 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:21:49.411532    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:21:55.608874 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:21:55.608837    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:22:25.609455 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:22:25.609415    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:22:49.896690 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:49.896656    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"]
Oct 30 04:22:49.897224 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:49.896902    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt" podUID="3cff5f9b-18b8-45b7-8052-09df41a8d91d" containerName="istio-proxy" containerID="cri-o://c7dc240dd32b78a1d2f6040935d7ccb202e068e54bb3ea13b0697a131e86514c" gracePeriod=30
Oct 30 04:22:55.152573 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.152542    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:22:55.248833 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.248791    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-podinfo\") pod \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") "
Oct 30 04:22:55.249015 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.248846    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-credential-socket\") pod \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") "
Oct 30 04:22:55.249015 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.248905    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cff5f9b-18b8-45b7-8052-09df41a8d91d-kube-api-access-qpcfm\" (UniqueName: \"kubernetes.io/projected/3cff5f9b-18b8-45b7-8052-09df41a8d91d-kube-api-access-qpcfm\") pod \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") "
Oct 30 04:22:55.249015 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.248929    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-envoy\") pod \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") "
Oct 30 04:22:55.249015 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.248949    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istiod-ca-cert\") pod \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") "
Oct 30 04:22:55.249015 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.248996    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-token\" (UniqueName: \"kubernetes.io/projected/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-token\") pod \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") "
Oct 30 04:22:55.249316 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.249054    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-workload-socket\") pod \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") "
Oct 30 04:22:55.249316 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.249101    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-data\") pod \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") "
Oct 30 04:22:55.249316 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.249146    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-credential-socket" pod "3cff5f9b-18b8-45b7-8052-09df41a8d91d" (UID: "3cff5f9b-18b8-45b7-8052-09df41a8d91d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:22:55.249316 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.249204    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-workload-certs\") pod \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\" (UID: \"3cff5f9b-18b8-45b7-8052-09df41a8d91d\") "
Oct 30 04:22:55.249528 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.249409    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-workload-socket" pod "3cff5f9b-18b8-45b7-8052-09df41a8d91d" (UID: "3cff5f9b-18b8-45b7-8052-09df41a8d91d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:22:55.249906 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.249617    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-data" pod "3cff5f9b-18b8-45b7-8052-09df41a8d91d" (UID: "3cff5f9b-18b8-45b7-8052-09df41a8d91d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:22:55.249906 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.249650    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istiod-ca-cert" pod "3cff5f9b-18b8-45b7-8052-09df41a8d91d" (UID: "3cff5f9b-18b8-45b7-8052-09df41a8d91d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:22:55.249906 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.249774    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:22:55.249906 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.249793    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:22:55.249906 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.249806    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:22:55.249906 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.249819    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:22:55.249906 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.249857    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-workload-certs" pod "3cff5f9b-18b8-45b7-8052-09df41a8d91d" (UID: "3cff5f9b-18b8-45b7-8052-09df41a8d91d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:22:55.251102 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.251076    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-podinfo" pod "3cff5f9b-18b8-45b7-8052-09df41a8d91d" (UID: "3cff5f9b-18b8-45b7-8052-09df41a8d91d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:22:55.251564 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.251536    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-envoy" pod "3cff5f9b-18b8-45b7-8052-09df41a8d91d" (UID: "3cff5f9b-18b8-45b7-8052-09df41a8d91d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:22:55.251789 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.251760    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-token" pod "3cff5f9b-18b8-45b7-8052-09df41a8d91d" (UID: "3cff5f9b-18b8-45b7-8052-09df41a8d91d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:22:55.252475 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.252448    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cff5f9b-18b8-45b7-8052-09df41a8d91d-kube-api-access-qpcfm" pod "3cff5f9b-18b8-45b7-8052-09df41a8d91d" (UID: "3cff5f9b-18b8-45b7-8052-09df41a8d91d"). InnerVolumeSpecName "kube-api-access-qpcfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:22:55.353443 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.353406    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:22:55.353443 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.353441    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:22:55.353443 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.353453    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpcfm\" (UniqueName: \"kubernetes.io/projected/3cff5f9b-18b8-45b7-8052-09df41a8d91d-kube-api-access-qpcfm\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:22:55.353756 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.353467    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:22:55.353756 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.353480    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cff5f9b-18b8-45b7-8052-09df41a8d91d-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:22:55.609137 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:22:55.609056    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:22:55.872818 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.872734    4874 generic.go:356] "Generic (PLEG): container finished" podID="3cff5f9b-18b8-45b7-8052-09df41a8d91d" containerID="c7dc240dd32b78a1d2f6040935d7ccb202e068e54bb3ea13b0697a131e86514c" exitCode=0
Oct 30 04:22:55.872818 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.872789    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"
Oct 30 04:22:55.873023 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.872816    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt" event={"ID":"3cff5f9b-18b8-45b7-8052-09df41a8d91d","Type":"ContainerDied","Data":"c7dc240dd32b78a1d2f6040935d7ccb202e068e54bb3ea13b0697a131e86514c"}
Oct 30 04:22:55.873023 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.872881    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt" event={"ID":"3cff5f9b-18b8-45b7-8052-09df41a8d91d","Type":"ContainerDied","Data":"a925cfc11520e46b1856d2e0fe3877d4183a7d493806a6a4a83953c4859d46e2"}
Oct 30 04:22:55.873023 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.872906    4874 scope.go:122] "RemoveContainer" containerID="c7dc240dd32b78a1d2f6040935d7ccb202e068e54bb3ea13b0697a131e86514c"
Oct 30 04:22:55.882950 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.882929    4874 scope.go:122] "RemoveContainer" containerID="c7dc240dd32b78a1d2f6040935d7ccb202e068e54bb3ea13b0697a131e86514c"
Oct 30 04:22:55.883195 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:22:55.883176    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7dc240dd32b78a1d2f6040935d7ccb202e068e54bb3ea13b0697a131e86514c\": container with ID starting with c7dc240dd32b78a1d2f6040935d7ccb202e068e54bb3ea13b0697a131e86514c not found: ID does not exist" containerID="c7dc240dd32b78a1d2f6040935d7ccb202e068e54bb3ea13b0697a131e86514c"
Oct 30 04:22:55.883304 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.883204    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7dc240dd32b78a1d2f6040935d7ccb202e068e54bb3ea13b0697a131e86514c"} err="failed to get container status \"c7dc240dd32b78a1d2f6040935d7ccb202e068e54bb3ea13b0697a131e86514c\": rpc error: code = NotFound desc = could not find container \"c7dc240dd32b78a1d2f6040935d7ccb202e068e54bb3ea13b0697a131e86514c\": container with ID starting with c7dc240dd32b78a1d2f6040935d7ccb202e068e54bb3ea13b0697a131e86514c not found: ID does not exist"
Oct 30 04:22:55.939077 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.939050    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"]
Oct 30 04:22:55.941845 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:55.941821    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sdbgt"]
Oct 30 04:22:56.729513 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:22:56.729484    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cff5f9b-18b8-45b7-8052-09df41a8d91d" path="/var/lib/kubelet/pods/3cff5f9b-18b8-45b7-8052-09df41a8d91d/volumes"
Oct 30 04:23:00.856207 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:00.856176    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 04:23:00.884497 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:00.884472    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 04:23:00.953675 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:00.953648    4874 scope.go:122] "RemoveContainer" containerID="8eab4bccb7cef8f06a4b7d1e228983de68469d70ad42775f8b53ae80cf740bd3"
Oct 30 04:23:24.294918 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:24.294885    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4"]
Oct 30 04:23:24.295398 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:24.295370    4874 cpu_manager.go:414] "containerMap: removing container" podUID="3cff5f9b-18b8-45b7-8052-09df41a8d91d" containerName="istio-proxy"
Oct 30 04:23:24.295398 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:24.295382    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cff5f9b-18b8-45b7-8052-09df41a8d91d" containerName="istio-proxy"
Oct 30 04:23:24.295527 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:24.295513    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cff5f9b-18b8-45b7-8052-09df41a8d91d" containerName="istio-proxy"
Oct 30 04:23:24.298678 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:24.298662    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4"
Oct 30 04:23:24.316128 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:24.316106    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1a0d3427-dockercfg-rxxdl\""
Oct 30 04:23:24.318820 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:24.318799    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4"]
Oct 30 04:23:24.328437 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:24.328411    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feabd4f1-bcc9-4529-9001-4580967be5d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4\" (UID: \"feabd4f1-bcc9-4529-9001-4580967be5d1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4"
Oct 30 04:23:24.328558 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:24.328512    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5xck\" (UniqueName: \"kubernetes.io/projected/feabd4f1-bcc9-4529-9001-4580967be5d1-kube-api-access-w5xck\") pod \"payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4\" (UID: \"feabd4f1-bcc9-4529-9001-4580967be5d1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4"
Oct 30 04:23:24.379408 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:24.379269    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1a0d3427\""
Oct 30 04:23:24.447569 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:24.447526    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feabd4f1-bcc9-4529-9001-4580967be5d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4\" (UID: \"feabd4f1-bcc9-4529-9001-4580967be5d1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4"
Oct 30 04:23:24.448888 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:24.448745    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feabd4f1-bcc9-4529-9001-4580967be5d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4\" (UID: \"feabd4f1-bcc9-4529-9001-4580967be5d1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4"
Oct 30 04:23:24.449428 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:24.448830    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5xck\" (UniqueName: \"kubernetes.io/projected/feabd4f1-bcc9-4529-9001-4580967be5d1-kube-api-access-w5xck\") pod \"payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4\" (UID: \"feabd4f1-bcc9-4529-9001-4580967be5d1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4"
Oct 30 04:23:24.474606 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:24.474577    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5xck\" (UniqueName: \"kubernetes.io/projected/feabd4f1-bcc9-4529-9001-4580967be5d1-kube-api-access-w5xck\") pod \"payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4\" (UID: \"feabd4f1-bcc9-4529-9001-4580967be5d1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4"
Oct 30 04:23:24.612068 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:24.611989    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4"
Oct 30 04:23:24.914430 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:24.914406    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4"]
Oct 30 04:23:24.916929 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:23:24.916901    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeabd4f1_bcc9_4529_9001_4580967be5d1.slice/crio-098b41f5d3b342dcc8dcc9093005b2f3a7d8ec9fbb642c61c52d9de8d952823f WatchSource:0}: Error finding container 098b41f5d3b342dcc8dcc9093005b2f3a7d8ec9fbb642c61c52d9de8d952823f: Status 404 returned error can't find the container with id 098b41f5d3b342dcc8dcc9093005b2f3a7d8ec9fbb642c61c52d9de8d952823f
Oct 30 04:23:25.026057 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:25.026020    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4" event={"ID":"feabd4f1-bcc9-4529-9001-4580967be5d1","Type":"ContainerStarted","Data":"968d2ef42fa2d6b2376facdfb0db64982b03da6cca2fab3ebcec854ae678613a"}
Oct 30 04:23:25.026194 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:25.026087    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4"
Oct 30 04:23:25.026194 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:25.026109    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4" event={"ID":"feabd4f1-bcc9-4529-9001-4580967be5d1","Type":"ContainerStarted","Data":"098b41f5d3b342dcc8dcc9093005b2f3a7d8ec9fbb642c61c52d9de8d952823f"}
Oct 30 04:23:25.064652 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:25.064580    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4" podStartSLOduration=1.064562936 podStartE2EDuration="1.064562936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:23:25.06396303 +0000 UTC m=+1524.476358602" watchObservedRunningTime="2026-10-30 04:23:25.064562936 +0000 UTC m=+1524.476958511"
Oct 30 04:23:25.609697 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:23:25.609663    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:23:35.099767 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:35.099736    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"]
Oct 30 04:23:35.100198 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:35.099945    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb" podUID="e22690dd-4871-45e2-b49e-8aaf73f19ae8" containerName="istio-proxy" containerID="cri-o://eb9d851a1fb28e63f92adb000881097ae7e61c5a376837b87e49725af40c65fe" gracePeriod=30
Oct 30 04:23:36.034983 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:36.034952    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4"
Oct 30 04:23:39.186206 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:39.186165    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:23:40.378655 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.378630    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:23:40.444899 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.444811    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istiod-ca-cert\") pod \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") "
Oct 30 04:23:40.444899 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.444882    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-workload-certs\") pod \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") "
Oct 30 04:23:40.445267 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.445206    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-workload-certs" pod "e22690dd-4871-45e2-b49e-8aaf73f19ae8" (UID: "e22690dd-4871-45e2-b49e-8aaf73f19ae8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:23:40.445408 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.445234    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istiod-ca-cert" pod "e22690dd-4871-45e2-b49e-8aaf73f19ae8" (UID: "e22690dd-4871-45e2-b49e-8aaf73f19ae8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:23:40.445576 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.445551    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-workload-socket\") pod \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") "
Oct 30 04:23:40.445686 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.445587    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-envoy\") pod \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") "
Oct 30 04:23:40.445686 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.445618    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-token\" (UniqueName: \"kubernetes.io/projected/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-token\") pod \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") "
Oct 30 04:23:40.445804 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.445718    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e22690dd-4871-45e2-b49e-8aaf73f19ae8-kube-api-access-5l8wr\" (UniqueName: \"kubernetes.io/projected/e22690dd-4871-45e2-b49e-8aaf73f19ae8-kube-api-access-5l8wr\") pod \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") "
Oct 30 04:23:40.445804 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.445789    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-podinfo\") pod \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") "
Oct 30 04:23:40.445908 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.445858    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-data\") pod \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") "
Oct 30 04:23:40.445908 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.445901    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-credential-socket\") pod \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\" (UID: \"e22690dd-4871-45e2-b49e-8aaf73f19ae8\") "
Oct 30 04:23:40.446440 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.446415    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:23:40.446533 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.446445    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:23:40.446875 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.446845    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-data" pod "e22690dd-4871-45e2-b49e-8aaf73f19ae8" (UID: "e22690dd-4871-45e2-b49e-8aaf73f19ae8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:23:40.447068 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.447046    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-workload-socket" pod "e22690dd-4871-45e2-b49e-8aaf73f19ae8" (UID: "e22690dd-4871-45e2-b49e-8aaf73f19ae8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:23:40.448736 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.448700    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-envoy" pod "e22690dd-4871-45e2-b49e-8aaf73f19ae8" (UID: "e22690dd-4871-45e2-b49e-8aaf73f19ae8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:23:40.449544 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.449522    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-podinfo" pod "e22690dd-4871-45e2-b49e-8aaf73f19ae8" (UID: "e22690dd-4871-45e2-b49e-8aaf73f19ae8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:23:40.449870 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.449824    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e22690dd-4871-45e2-b49e-8aaf73f19ae8-kube-api-access-5l8wr" pod "e22690dd-4871-45e2-b49e-8aaf73f19ae8" (UID: "e22690dd-4871-45e2-b49e-8aaf73f19ae8"). InnerVolumeSpecName "kube-api-access-5l8wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:23:40.450029 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.450006    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-credential-socket" pod "e22690dd-4871-45e2-b49e-8aaf73f19ae8" (UID: "e22690dd-4871-45e2-b49e-8aaf73f19ae8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:23:40.450753 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.450710    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-token" pod "e22690dd-4871-45e2-b49e-8aaf73f19ae8" (UID: "e22690dd-4871-45e2-b49e-8aaf73f19ae8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:23:40.553194 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.553057    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:23:40.553194 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.553095    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:23:40.553194 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.553111    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:23:40.553194 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.553125    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:23:40.553194 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.553140    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l8wr\" (UniqueName: \"kubernetes.io/projected/e22690dd-4871-45e2-b49e-8aaf73f19ae8-kube-api-access-5l8wr\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:23:40.553194 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.553154    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:23:40.553194 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:40.553168    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e22690dd-4871-45e2-b49e-8aaf73f19ae8-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:23:41.124314 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.124202    4874 generic.go:356] "Generic (PLEG): container finished" podID="e22690dd-4871-45e2-b49e-8aaf73f19ae8" containerID="eb9d851a1fb28e63f92adb000881097ae7e61c5a376837b87e49725af40c65fe" exitCode=0
Oct 30 04:23:41.124314 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.124289    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"
Oct 30 04:23:41.124314 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.124307    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb" event={"ID":"e22690dd-4871-45e2-b49e-8aaf73f19ae8","Type":"ContainerDied","Data":"eb9d851a1fb28e63f92adb000881097ae7e61c5a376837b87e49725af40c65fe"}
Oct 30 04:23:41.124578 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.124356    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb" event={"ID":"e22690dd-4871-45e2-b49e-8aaf73f19ae8","Type":"ContainerDied","Data":"55eea014dff3b8d8e395faf8310ea39bf78b1059d70377b0f88a06fecc8114e2"}
Oct 30 04:23:41.124578 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.124385    4874 scope.go:122] "RemoveContainer" containerID="eb9d851a1fb28e63f92adb000881097ae7e61c5a376837b87e49725af40c65fe"
Oct 30 04:23:41.141989 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.141966    4874 scope.go:122] "RemoveContainer" containerID="eb9d851a1fb28e63f92adb000881097ae7e61c5a376837b87e49725af40c65fe"
Oct 30 04:23:41.142560 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:23:41.142521    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb9d851a1fb28e63f92adb000881097ae7e61c5a376837b87e49725af40c65fe\": container with ID starting with eb9d851a1fb28e63f92adb000881097ae7e61c5a376837b87e49725af40c65fe not found: ID does not exist" containerID="eb9d851a1fb28e63f92adb000881097ae7e61c5a376837b87e49725af40c65fe"
Oct 30 04:23:41.142685 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.142564    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb9d851a1fb28e63f92adb000881097ae7e61c5a376837b87e49725af40c65fe"} err="failed to get container status \"eb9d851a1fb28e63f92adb000881097ae7e61c5a376837b87e49725af40c65fe\": rpc error: code = NotFound desc = could not find container \"eb9d851a1fb28e63f92adb000881097ae7e61c5a376837b87e49725af40c65fe\": container with ID starting with eb9d851a1fb28e63f92adb000881097ae7e61c5a376837b87e49725af40c65fe not found: ID does not exist"
Oct 30 04:23:41.194570 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.192215    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"]
Oct 30 04:23:41.203623 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.203589    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xfldb"]
Oct 30 04:23:41.321964 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.321924    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"]
Oct 30 04:23:41.322724 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.322696    4874 cpu_manager.go:414] "containerMap: removing container" podUID="e22690dd-4871-45e2-b49e-8aaf73f19ae8" containerName="istio-proxy"
Oct 30 04:23:41.322904 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.322886    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e22690dd-4871-45e2-b49e-8aaf73f19ae8" containerName="istio-proxy"
Oct 30 04:23:41.323173 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.323156    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="e22690dd-4871-45e2-b49e-8aaf73f19ae8" containerName="istio-proxy"
Oct 30 04:23:41.328254 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.328209    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.349991 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.349949    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e124dc-openshift-default-dockercfg-7sccw\""
Oct 30 04:23:41.351630 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.351607    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"]
Oct 30 04:23:41.464002 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.463906    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-envoy\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.464735 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.464706    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-data\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.464950 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.464932    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-token\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.465122 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.465100    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-podinfo\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.472265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.465277    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istiod-ca-cert\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.472658 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.472635    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-credential-socket\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.472868 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.472849    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-workload-socket\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.473059 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.473042    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h7vc\" (UniqueName: \"kubernetes.io/projected/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-kube-api-access-5h7vc\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.480398 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.480370    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-workload-certs\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.586270 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.584961    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-envoy\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.586270 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.585016    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-data\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.586270 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.585065    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-token\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.586270 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.585102    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-podinfo\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.586270 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.585133    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istiod-ca-cert\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.586270 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.585196    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-credential-socket\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.586270 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.585259    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-workload-socket\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.586270 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.585323    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h7vc\" (UniqueName: \"kubernetes.io/projected/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-kube-api-access-5h7vc\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.586270 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.585362    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-workload-certs\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.586270 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.585809    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-workload-certs\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.586973 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.586727    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istiod-ca-cert\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.587038 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.586977    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-data\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.591266 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.587871    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-credential-socket\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.591266 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.588382    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-workload-socket\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.597260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.594525    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-envoy\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.597260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.595050    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-podinfo\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.616741 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.615818    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-token\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.617144 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.617073    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h7vc\" (UniqueName: \"kubernetes.io/projected/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-kube-api-access-5h7vc\") pod \"e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") " pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:41.649759 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:41.649732    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:42.044864 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:42.044819    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"]
Oct 30 04:23:42.054500 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:42.054460    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:23:42.054635 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:42.054531    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:23:42.054635 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:42.054569    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:23:42.136206 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:42.136170    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42" event={"ID":"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d","Type":"ContainerStarted","Data":"ad18e184286cb701f009df217f4d7d3be4b49a6b04579cc50ed46f720da83e1b"}
Oct 30 04:23:42.136386 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:42.136227    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42" event={"ID":"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d","Type":"ContainerStarted","Data":"e648dad55fd852f3200e23a4eaf30f0961d9232a21d77896d975088be8ff4c68"}
Oct 30 04:23:42.191261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:42.190458    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42" podStartSLOduration=1.190437574 podStartE2EDuration="1.190437574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:23:42.183674332 +0000 UTC m=+1541.596069907" watchObservedRunningTime="2026-10-30 04:23:42.190437574 +0000 UTC m=+1541.602833149"
Oct 30 04:23:42.650344 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:42.650314    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:42.742954 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:42.742915    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e22690dd-4871-45e2-b49e-8aaf73f19ae8" path="/var/lib/kubelet/pods/e22690dd-4871-45e2-b49e-8aaf73f19ae8/volumes"
Oct 30 04:23:43.657160 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:43.657131    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:44.147804 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:44.147767    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:44.148661 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:44.148638    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:23:55.608908 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:23:55.608876    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:23:57.382744 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:57.382712    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch"]
Oct 30 04:23:57.386792 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:57.386768    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch"
Oct 30 04:23:57.416007 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:57.415981    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e124dc-serving-cert\""
Oct 30 04:23:57.420427 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:57.420396    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch"]
Oct 30 04:23:57.492935 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:57.492904    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-openshift-service-ca\") pod \"maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch\" (UID: \"65809ef9-6fc6-4dd9-b224-0881ea5cad6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch"
Oct 30 04:23:57.493087 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:57.492955    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8blct\" (UniqueName: \"kubernetes.io/projected/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-kube-api-access-8blct\") pod \"maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch\" (UID: \"65809ef9-6fc6-4dd9-b224-0881ea5cad6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch"
Oct 30 04:23:57.493087 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:57.493069    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-maas-api-tls\") pod \"maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch\" (UID: \"65809ef9-6fc6-4dd9-b224-0881ea5cad6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch"
Oct 30 04:23:57.493158 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:57.493098    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-metrics-certs\") pod \"maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch\" (UID: \"65809ef9-6fc6-4dd9-b224-0881ea5cad6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch"
Oct 30 04:23:57.594749 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:57.594703    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-openshift-service-ca\") pod \"maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch\" (UID: \"65809ef9-6fc6-4dd9-b224-0881ea5cad6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch"
Oct 30 04:23:57.594906 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:57.594773    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8blct\" (UniqueName: \"kubernetes.io/projected/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-kube-api-access-8blct\") pod \"maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch\" (UID: \"65809ef9-6fc6-4dd9-b224-0881ea5cad6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch"
Oct 30 04:23:57.594906 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:57.594824    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-maas-api-tls\") pod \"maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch\" (UID: \"65809ef9-6fc6-4dd9-b224-0881ea5cad6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch"
Oct 30 04:23:57.594906 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:57.594845    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-metrics-certs\") pod \"maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch\" (UID: \"65809ef9-6fc6-4dd9-b224-0881ea5cad6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch"
Oct 30 04:23:57.595475 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:57.595453    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-openshift-service-ca\") pod \"maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch\" (UID: \"65809ef9-6fc6-4dd9-b224-0881ea5cad6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch"
Oct 30 04:23:57.597308 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:57.597286    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-maas-api-tls\") pod \"maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch\" (UID: \"65809ef9-6fc6-4dd9-b224-0881ea5cad6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch"
Oct 30 04:23:57.597391 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:57.597315    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-metrics-certs\") pod \"maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch\" (UID: \"65809ef9-6fc6-4dd9-b224-0881ea5cad6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch"
Oct 30 04:23:57.617311 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:57.617286    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8blct\" (UniqueName: \"kubernetes.io/projected/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-kube-api-access-8blct\") pod \"maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch\" (UID: \"65809ef9-6fc6-4dd9-b224-0881ea5cad6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch"
Oct 30 04:23:57.705283 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:57.703416    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch"
Oct 30 04:23:58.045612 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:58.045589    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch"]
Oct 30 04:23:58.047684 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:23:58.047660    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65809ef9_6fc6_4dd9_b224_0881ea5cad6c.slice/crio-5d392640679f9efaaa1aff3a5d356c51581ef862b68b9875d755ece13852a5d3 WatchSource:0}: Error finding container 5d392640679f9efaaa1aff3a5d356c51581ef862b68b9875d755ece13852a5d3: Status 404 returned error can't find the container with id 5d392640679f9efaaa1aff3a5d356c51581ef862b68b9875d755ece13852a5d3
Oct 30 04:23:58.049121 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:58.049104    4874 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 04:23:58.222302 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:58.222266    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch" event={"ID":"65809ef9-6fc6-4dd9-b224-0881ea5cad6c","Type":"ContainerStarted","Data":"5d392640679f9efaaa1aff3a5d356c51581ef862b68b9875d755ece13852a5d3"}
Oct 30 04:23:59.252264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:59.252194    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch" event={"ID":"65809ef9-6fc6-4dd9-b224-0881ea5cad6c","Type":"ContainerStarted","Data":"3e4bb00bc4cc27d38c5ff9261c581dc4ec99cc959155be12707f11c9d21b532b"}
Oct 30 04:23:59.253281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:59.253253    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch"
Oct 30 04:23:59.295401 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:23:59.294929    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch" podStartSLOduration=1.818061242 podStartE2EDuration="2.294911693s" totalImagesPullingTime="476.850451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:23:59.294065389 +0000 UTC m=+1558.706460961" watchObservedRunningTime="2026-10-30 04:23:59.294911693 +0000 UTC m=+1558.707307272"
Oct 30 04:24:00.349305 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:00.349268    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:24:05.201556 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.201522    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"]
Oct 30 04:24:05.206662 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.206638    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.234139 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.234112    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"]
Oct 30 04:24:05.235605 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.235576    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/908b541f-e807-4403-98c4-41e5cea32837-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.235750 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.235626    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.235820 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.235762    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908b541f-e807-4403-98c4-41e5cea32837-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.235880 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.235849    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.235941 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.235881    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.235941 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.235919    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.236660 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.235952    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4q87\" (UniqueName: \"kubernetes.io/projected/908b541f-e807-4403-98c4-41e5cea32837-kube-api-access-q4q87\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.236660 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.236000    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.236660 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.236024    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/908b541f-e807-4403-98c4-41e5cea32837-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.337187 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.337155    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.337187 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.337191    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.337431 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.337226    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.337431 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.337283    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4q87\" (UniqueName: \"kubernetes.io/projected/908b541f-e807-4403-98c4-41e5cea32837-kube-api-access-q4q87\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.337431 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.337333    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.337431 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.337357    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/908b541f-e807-4403-98c4-41e5cea32837-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.337650 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.337473    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/908b541f-e807-4403-98c4-41e5cea32837-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.337650 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.337509    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.337650 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.337582    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908b541f-e807-4403-98c4-41e5cea32837-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.337650 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.337642    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.337831 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.337718    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.337994 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.337972    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.338053 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.338012    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.338297 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.338272    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/908b541f-e807-4403-98c4-41e5cea32837-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.339664 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.339644    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.339981 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.339961    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/908b541f-e807-4403-98c4-41e5cea32837-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.359587 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.359560    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908b541f-e807-4403-98c4-41e5cea32837-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.359722 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.359700    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4q87\" (UniqueName: \"kubernetes.io/projected/908b541f-e807-4403-98c4-41e5cea32837-kube-api-access-q4q87\") pod \"maas-default-gateway-openshift-default-7f694df8bc-czkn2\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.520109 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.520083    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:05.840906 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.840882    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"]
Oct 30 04:24:05.843769 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:24:05.843741    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908b541f_e807_4403_98c4_41e5cea32837.slice/crio-7b5923152c1233164c57c89fc9369e677313ce8d8a1f1b0bea311d010054c4a7 WatchSource:0}: Error finding container 7b5923152c1233164c57c89fc9369e677313ce8d8a1f1b0bea311d010054c4a7: Status 404 returned error can't find the container with id 7b5923152c1233164c57c89fc9369e677313ce8d8a1f1b0bea311d010054c4a7
Oct 30 04:24:05.846018 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.845989    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:24:05.846090 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.846050    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:24:05.846090 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:05.846078    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:24:06.306466 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:06.306433    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch"
Oct 30 04:24:06.307970 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:06.307933    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2" event={"ID":"908b541f-e807-4403-98c4-41e5cea32837","Type":"ContainerStarted","Data":"3869f7e77e1916369207bd75eb0ef2b7e0db8adb777474de44bfa3fe30828802"}
Oct 30 04:24:06.308108 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:06.307986    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2" event={"ID":"908b541f-e807-4403-98c4-41e5cea32837","Type":"ContainerStarted","Data":"7b5923152c1233164c57c89fc9369e677313ce8d8a1f1b0bea311d010054c4a7"}
Oct 30 04:24:06.403482 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:06.403419    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2" podStartSLOduration=1.4034011 podStartE2EDuration="1.4034011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:24:06.396095817 +0000 UTC m=+1565.808491389" watchObservedRunningTime="2026-10-30 04:24:06.4034011 +0000 UTC m=+1565.815796677"
Oct 30 04:24:06.520762 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:06.520721    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:06.522401 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:06.522377    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-czkn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 04:24:06.522533 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:06.522424    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2" podUID="908b541f-e807-4403-98c4-41e5cea32837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 04:24:07.522694 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:07.522653    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-czkn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 04:24:07.523268 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:07.523215    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2" podUID="908b541f-e807-4403-98c4-41e5cea32837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 04:24:08.520630 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:08.520592    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-czkn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 04:24:08.520804 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:08.520660    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2" podUID="908b541f-e807-4403-98c4-41e5cea32837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 04:24:09.521498 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:09.521449    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-czkn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 04:24:09.521910 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:09.521522    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2" podUID="908b541f-e807-4403-98c4-41e5cea32837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 04:24:10.521380 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:10.521345    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-czkn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 04:24:10.521560 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:10.521399    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2" podUID="908b541f-e807-4403-98c4-41e5cea32837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 04:24:11.520532 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:11.520498    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-czkn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 04:24:11.520712 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:11.520551    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2" podUID="908b541f-e807-4403-98c4-41e5cea32837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 04:24:12.521060 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:12.521025    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-czkn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 04:24:12.521481 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:12.521080    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2" podUID="908b541f-e807-4403-98c4-41e5cea32837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 04:24:13.046311 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:13.035764    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt"]
Oct 30 04:24:13.064279 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:13.062408    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt"
Oct 30 04:24:13.076266 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:13.075981    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d36063-1391-428f-955b-5a6d6f4cba1b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt\" (UID: \"75d36063-1391-428f-955b-5a6d6f4cba1b\") " pod="openshift-ingress/payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt"
Oct 30 04:24:13.076497 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:13.076393    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwzw9\" (UniqueName: \"kubernetes.io/projected/75d36063-1391-428f-955b-5a6d6f4cba1b-kube-api-access-jwzw9\") pod \"payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt\" (UID: \"75d36063-1391-428f-955b-5a6d6f4cba1b\") " pod="openshift-ingress/payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt"
Oct 30 04:24:13.091461 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:13.089603    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt"]
Oct 30 04:24:13.114353 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:13.112218    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e3c43b\""
Oct 30 04:24:13.155365 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:13.138204    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e3c43b-dockercfg-p8tqp\""
Oct 30 04:24:13.191516 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:13.187967    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwzw9\" (UniqueName: \"kubernetes.io/projected/75d36063-1391-428f-955b-5a6d6f4cba1b-kube-api-access-jwzw9\") pod \"payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt\" (UID: \"75d36063-1391-428f-955b-5a6d6f4cba1b\") " pod="openshift-ingress/payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt"
Oct 30 04:24:13.191516 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:13.188114    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d36063-1391-428f-955b-5a6d6f4cba1b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt\" (UID: \"75d36063-1391-428f-955b-5a6d6f4cba1b\") " pod="openshift-ingress/payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt"
Oct 30 04:24:13.221260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:13.220736    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d36063-1391-428f-955b-5a6d6f4cba1b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt\" (UID: \"75d36063-1391-428f-955b-5a6d6f4cba1b\") " pod="openshift-ingress/payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt"
Oct 30 04:24:13.307785 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:13.307512    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwzw9\" (UniqueName: \"kubernetes.io/projected/75d36063-1391-428f-955b-5a6d6f4cba1b-kube-api-access-jwzw9\") pod \"payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt\" (UID: \"75d36063-1391-428f-955b-5a6d6f4cba1b\") " pod="openshift-ingress/payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt"
Oct 30 04:24:13.499354 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:13.499317    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt"
Oct 30 04:24:13.520789 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:13.520759    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-czkn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 04:24:13.520909 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:13.520818    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2" podUID="908b541f-e807-4403-98c4-41e5cea32837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 04:24:13.904829 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:13.904802    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt"]
Oct 30 04:24:13.907405 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:24:13.907376    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d36063_1391_428f_955b_5a6d6f4cba1b.slice/crio-4c714222ba3554bfdd4b5813e1ed800d613b5af595d6e09b63c898d1ef85edb1 WatchSource:0}: Error finding container 4c714222ba3554bfdd4b5813e1ed800d613b5af595d6e09b63c898d1ef85edb1: Status 404 returned error can't find the container with id 4c714222ba3554bfdd4b5813e1ed800d613b5af595d6e09b63c898d1ef85edb1
Oct 30 04:24:14.431961 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:14.431920    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt" event={"ID":"75d36063-1391-428f-955b-5a6d6f4cba1b","Type":"ContainerStarted","Data":"346ae8f05a18bf6e72fec6289946b4afb66b738582c4958df378c06bad7997fb"}
Oct 30 04:24:14.432162 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:14.431974    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt" event={"ID":"75d36063-1391-428f-955b-5a6d6f4cba1b","Type":"ContainerStarted","Data":"4c714222ba3554bfdd4b5813e1ed800d613b5af595d6e09b63c898d1ef85edb1"}
Oct 30 04:24:14.432162 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:14.431996    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt"
Oct 30 04:24:14.478262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:14.475186    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt" podStartSLOduration=2.475170052 podStartE2EDuration="2.475170052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:24:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:24:14.473453092 +0000 UTC m=+1573.885848667" watchObservedRunningTime="2026-10-30 04:24:14.475170052 +0000 UTC m=+1573.887565626"
Oct 30 04:24:14.521448 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:14.521417    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-czkn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 04:24:14.521583 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:14.521465    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2" podUID="908b541f-e807-4403-98c4-41e5cea32837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 04:24:15.909071 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:15.909033    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:15.910084 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:15.910064    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:15.910662 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:15.910643    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:24:16.198696 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.198605    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"]
Oct 30 04:24:16.202940 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.202916    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.218799 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.218777    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0c530e-openshift-default-dockercfg-jnw6l\""
Oct 30 04:24:16.224334 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.224311    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"]
Oct 30 04:24:16.258938 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.258910    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-workload-certs\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.259074 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.258969    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnq5w\" (UniqueName: \"kubernetes.io/projected/32b4cf74-389e-4749-bf55-a1c52313e609-kube-api-access-pnq5w\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.259074 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.259049    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-istio-envoy\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.259176 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.259093    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-credential-socket\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.259284 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.259222    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b4cf74-389e-4749-bf55-a1c52313e609-istio-token\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.259284 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.259267    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32b4cf74-389e-4749-bf55-a1c52313e609-istiod-ca-cert\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.259409 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.259351    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-workload-socket\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.259409 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.259379    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-istio-data\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.259536 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.259519    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32b4cf74-389e-4749-bf55-a1c52313e609-istio-podinfo\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.360559 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.360530    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32b4cf74-389e-4749-bf55-a1c52313e609-istio-podinfo\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.360716 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.360587    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-workload-certs\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.360716 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.360627    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnq5w\" (UniqueName: \"kubernetes.io/projected/32b4cf74-389e-4749-bf55-a1c52313e609-kube-api-access-pnq5w\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.360716 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.360647    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-istio-envoy\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.360716 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.360683    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-credential-socket\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.360892 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.360744    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b4cf74-389e-4749-bf55-a1c52313e609-istio-token\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.360892 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.360768    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32b4cf74-389e-4749-bf55-a1c52313e609-istiod-ca-cert\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.360892 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.360807    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-workload-socket\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.360892 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.360831    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-istio-data\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.361095 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.361079    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-workload-certs\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.361184 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.361145    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-credential-socket\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.361271 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.361186    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-istio-data\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.361338 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.361267    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-workload-socket\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.361565 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.361539    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32b4cf74-389e-4749-bf55-a1c52313e609-istiod-ca-cert\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.363143 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.363122    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32b4cf74-389e-4749-bf55-a1c52313e609-istio-podinfo\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.363313 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.363295    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-istio-envoy\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.391794 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.391744    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b4cf74-389e-4749-bf55-a1c52313e609-istio-token\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.392299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.392099    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnq5w\" (UniqueName: \"kubernetes.io/projected/32b4cf74-389e-4749-bf55-a1c52313e609-kube-api-access-pnq5w\") pod \"e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") " pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.523783 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.523489    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:16.910163 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.910141    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"]
Oct 30 04:24:16.912436 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:24:16.912406    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b4cf74_389e_4749_bf55_a1c52313e609.slice/crio-5506d485dc0a165259f654d959f50e20684d6bbf5453de08e8382a21cc7babc5 WatchSource:0}: Error finding container 5506d485dc0a165259f654d959f50e20684d6bbf5453de08e8382a21cc7babc5: Status 404 returned error can't find the container with id 5506d485dc0a165259f654d959f50e20684d6bbf5453de08e8382a21cc7babc5
Oct 30 04:24:16.914760 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.914729    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:24:16.914842 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.914793    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:24:16.914842 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:16.914822    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:24:17.494403 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:17.494360    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh" event={"ID":"32b4cf74-389e-4749-bf55-a1c52313e609","Type":"ContainerStarted","Data":"eb093c1b82de3a92d2c5265b0ef0d70c2ffe1e5073602d271324afc37aa33a5a"}
Oct 30 04:24:17.494598 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:17.494426    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh" event={"ID":"32b4cf74-389e-4749-bf55-a1c52313e609","Type":"ContainerStarted","Data":"5506d485dc0a165259f654d959f50e20684d6bbf5453de08e8382a21cc7babc5"}
Oct 30 04:24:17.524723 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:17.524695    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:17.530190 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:17.530166    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:17.540006 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:17.539963    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh" podStartSLOduration=1.5399502630000002 podStartE2EDuration="1.539950263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:24:17.53746667 +0000 UTC m=+1576.949862242" watchObservedRunningTime="2026-10-30 04:24:17.539950263 +0000 UTC m=+1576.952345835"
Oct 30 04:24:18.500684 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:18.500652    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:18.501617 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:18.501596    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:24:20.265314 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.265279    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"]
Oct 30 04:24:20.268979 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.268956    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.302671 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.302642    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"]
Oct 30 04:24:20.338145 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.338106    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.338145 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.338143    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.338390 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.338172    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.338390 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.338282    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157dc9a9-a3f1-4de0-9787-22207838b7b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.338390 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.338325    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hstc\" (UniqueName: \"kubernetes.io/projected/157dc9a9-a3f1-4de0-9787-22207838b7b7-kube-api-access-5hstc\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.338558 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.338461    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.338626 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.338606    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.338688 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.338669    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.338793 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.338771    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.440138 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.440105    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.440299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.440145    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.440299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.440187    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.440299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.440232    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.440435 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.440310    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157dc9a9-a3f1-4de0-9787-22207838b7b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.440435 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.440333    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hstc\" (UniqueName: \"kubernetes.io/projected/157dc9a9-a3f1-4de0-9787-22207838b7b7-kube-api-access-5hstc\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.440435 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.440411    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.440585 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.440531    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.440643 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.440593    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.440643 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.440631    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.440756 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.440669    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.440930 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.440905    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.441014 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.440974    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.441130 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.441108    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157dc9a9-a3f1-4de0-9787-22207838b7b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.442675 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.442646    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.442912 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.442894    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.477909 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.477885    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.478807 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.478786    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hstc\" (UniqueName: \"kubernetes.io/projected/157dc9a9-a3f1-4de0-9787-22207838b7b7-kube-api-access-5hstc\") pod \"maas-default-gateway-openshift-default-7f694df8bc-q4qrg\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.584392 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.584315    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:20.910691 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.910669    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"]
Oct 30 04:24:20.913377 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:24:20.913349    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157dc9a9_a3f1_4de0_9787_22207838b7b7.slice/crio-5b5421833629918ca5c2a57d5413280a6fafa0f460014ac2e7135c18dedb24d6 WatchSource:0}: Error finding container 5b5421833629918ca5c2a57d5413280a6fafa0f460014ac2e7135c18dedb24d6: Status 404 returned error can't find the container with id 5b5421833629918ca5c2a57d5413280a6fafa0f460014ac2e7135c18dedb24d6
Oct 30 04:24:20.915842 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.915811    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:24:20.915944 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.915886    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:24:20.915944 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:20.915929    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:24:21.520903 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:21.520872    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg" event={"ID":"157dc9a9-a3f1-4de0-9787-22207838b7b7","Type":"ContainerStarted","Data":"7d7d7e235b43707b4c5ab8e8aca9330cc6443e91cec5ce978e6599eb16b2696e"}
Oct 30 04:24:21.521313 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:21.520925    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg" event={"ID":"157dc9a9-a3f1-4de0-9787-22207838b7b7","Type":"ContainerStarted","Data":"5b5421833629918ca5c2a57d5413280a6fafa0f460014ac2e7135c18dedb24d6"}
Oct 30 04:24:21.584716 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:21.584688    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:21.595871 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:21.595823    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg" podStartSLOduration=1.595807599 podStartE2EDuration="1.595807599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:24:21.582510119 +0000 UTC m=+1580.994905693" watchObservedRunningTime="2026-10-30 04:24:21.595807599 +0000 UTC m=+1581.008203173"
Oct 30 04:24:22.527684 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:22.527651    4874 generic.go:356] "Generic (PLEG): container finished" podID="f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" containerID="ad18e184286cb701f009df217f4d7d3be4b49a6b04579cc50ed46f720da83e1b" exitCode=0
Oct 30 04:24:22.528180 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:22.527720    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42" event={"ID":"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d","Type":"ContainerDied","Data":"ad18e184286cb701f009df217f4d7d3be4b49a6b04579cc50ed46f720da83e1b"}
Oct 30 04:24:22.528390 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:22.528370    4874 scope.go:122] "RemoveContainer" containerID="ad18e184286cb701f009df217f4d7d3be4b49a6b04579cc50ed46f720da83e1b"
Oct 30 04:24:22.529687 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:22.529660    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:24:22.529800 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:22.529741    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:24:22.529800 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:22.529790    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:24:22.594296 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:22.593374    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-q4qrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 04:24:22.594296 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:22.593433    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg" podUID="157dc9a9-a3f1-4de0-9787-22207838b7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 04:24:23.231515 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:23.231479    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:24:23.536328 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:23.536225    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42" event={"ID":"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d","Type":"ContainerStarted","Data":"1d8143bdcb14ed4d08d7da80b45e6f470bde7c97362d69d5235fe50ac5654eaf"}
Oct 30 04:24:23.588265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:23.586623    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-q4qrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 04:24:23.588265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:23.586683    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg" podUID="157dc9a9-a3f1-4de0-9787-22207838b7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 04:24:23.650149 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:23.650117    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:24:23.650439 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:23.650416    4874 patch_prober.go:29] interesting pod/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 04:24:23.650560 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:23.650468    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42" podUID="f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 04:24:24.587724 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:24.587679    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-q4qrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 04:24:24.588467 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:24.588427    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg" podUID="157dc9a9-a3f1-4de0-9787-22207838b7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 04:24:24.650707 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:24.650658    4874 patch_prober.go:29] interesting pod/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 04:24:24.650906 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:24.650723    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42" podUID="f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 04:24:25.441838 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:25.441798    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt"
Oct 30 04:24:25.588856 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:25.588820    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-q4qrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 04:24:25.594514 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:25.594465    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg" podUID="157dc9a9-a3f1-4de0-9787-22207838b7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 04:24:25.650299 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:24:25.650252    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:24:25.651310 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:25.651282    4874 patch_prober.go:29] interesting pod/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 04:24:25.651422 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:25.651335    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42" podUID="f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 04:24:26.586622 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:26.586587    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-q4qrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 04:24:26.586903 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:26.586873    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg" podUID="157dc9a9-a3f1-4de0-9787-22207838b7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 04:24:26.650705 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:26.650675    4874 patch_prober.go:29] interesting pod/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 04:24:26.651119 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:26.651094    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42" podUID="f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 04:24:27.586048 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:27.585613    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-q4qrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 04:24:27.586048 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:27.585672    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg" podUID="157dc9a9-a3f1-4de0-9787-22207838b7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 04:24:27.650699 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:27.650666    4874 patch_prober.go:29] interesting pod/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 04:24:27.650851 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:27.650715    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42" podUID="f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 04:24:28.585979 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:28.585570    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-q4qrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 04:24:28.585979 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:28.585630    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg" podUID="157dc9a9-a3f1-4de0-9787-22207838b7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 04:24:28.650712 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:28.650675    4874 patch_prober.go:29] interesting pod/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 04:24:28.650889 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:28.650718    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42" podUID="f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 04:24:29.147705 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:29.147672    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:24:29.655355 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:29.655324    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:24:29.656537 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:29.656516    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:24:30.013894 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:30.013864    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:30.014185 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:30.014156    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:30.014996 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:30.014976    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:24:34.886153 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:34.886121    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8"]
Oct 30 04:24:34.891137 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:34.891118    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8"
Oct 30 04:24:34.918926 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:34.918901    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0c530e-serving-cert\""
Oct 30 04:24:34.923386 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:34.923361    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8"]
Oct 30 04:24:34.945306 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:34.945275    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c53e382-3a2d-438a-8093-aef7966c1913-metrics-certs\") pod \"maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8\" (UID: \"9c53e382-3a2d-438a-8093-aef7966c1913\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8"
Oct 30 04:24:34.945467 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:34.945371    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c53e382-3a2d-438a-8093-aef7966c1913-maas-api-tls\") pod \"maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8\" (UID: \"9c53e382-3a2d-438a-8093-aef7966c1913\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8"
Oct 30 04:24:34.945467 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:34.945400    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c53e382-3a2d-438a-8093-aef7966c1913-openshift-service-ca\") pod \"maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8\" (UID: \"9c53e382-3a2d-438a-8093-aef7966c1913\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8"
Oct 30 04:24:34.945625 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:34.945601    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qnd9\" (UniqueName: \"kubernetes.io/projected/9c53e382-3a2d-438a-8093-aef7966c1913-kube-api-access-2qnd9\") pod \"maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8\" (UID: \"9c53e382-3a2d-438a-8093-aef7966c1913\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8"
Oct 30 04:24:35.046652 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:35.046624    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qnd9\" (UniqueName: \"kubernetes.io/projected/9c53e382-3a2d-438a-8093-aef7966c1913-kube-api-access-2qnd9\") pod \"maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8\" (UID: \"9c53e382-3a2d-438a-8093-aef7966c1913\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8"
Oct 30 04:24:35.046814 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:35.046679    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c53e382-3a2d-438a-8093-aef7966c1913-metrics-certs\") pod \"maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8\" (UID: \"9c53e382-3a2d-438a-8093-aef7966c1913\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8"
Oct 30 04:24:35.046814 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:35.046743    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c53e382-3a2d-438a-8093-aef7966c1913-maas-api-tls\") pod \"maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8\" (UID: \"9c53e382-3a2d-438a-8093-aef7966c1913\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8"
Oct 30 04:24:35.046814 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:35.046768    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c53e382-3a2d-438a-8093-aef7966c1913-openshift-service-ca\") pod \"maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8\" (UID: \"9c53e382-3a2d-438a-8093-aef7966c1913\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8"
Oct 30 04:24:35.046972 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:24:35.046832    4874 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-serving-cert: secret "maas-api-e2e-api-b-0c530e-serving-cert" not found
Oct 30 04:24:35.046972 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:24:35.046904    4874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c53e382-3a2d-438a-8093-aef7966c1913-maas-api-tls podName:9c53e382-3a2d-438a-8093-aef7966c1913 nodeName:}" failed. No retries permitted until 2026-10-30 04:24:35.546888181 +0000 UTC m=+1594.959283736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9c53e382-3a2d-438a-8093-aef7966c1913-maas-api-tls") pod "maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8" (UID: "9c53e382-3a2d-438a-8093-aef7966c1913") : secret "maas-api-e2e-api-b-0c530e-serving-cert" not found
Oct 30 04:24:35.047458 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:35.047436    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c53e382-3a2d-438a-8093-aef7966c1913-openshift-service-ca\") pod \"maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8\" (UID: \"9c53e382-3a2d-438a-8093-aef7966c1913\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8"
Oct 30 04:24:35.049001 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:35.048983    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c53e382-3a2d-438a-8093-aef7966c1913-metrics-certs\") pod \"maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8\" (UID: \"9c53e382-3a2d-438a-8093-aef7966c1913\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8"
Oct 30 04:24:35.069494 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:35.069471    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qnd9\" (UniqueName: \"kubernetes.io/projected/9c53e382-3a2d-438a-8093-aef7966c1913-kube-api-access-2qnd9\") pod \"maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8\" (UID: \"9c53e382-3a2d-438a-8093-aef7966c1913\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8"
Oct 30 04:24:35.559554 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:35.559497    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c53e382-3a2d-438a-8093-aef7966c1913-maas-api-tls\") pod \"maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8\" (UID: \"9c53e382-3a2d-438a-8093-aef7966c1913\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8"
Oct 30 04:24:35.561891 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:35.561872    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c53e382-3a2d-438a-8093-aef7966c1913-maas-api-tls\") pod \"maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8\" (UID: \"9c53e382-3a2d-438a-8093-aef7966c1913\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8"
Oct 30 04:24:35.805190 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:35.805134    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8"
Oct 30 04:24:36.164011 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:36.163924    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8"]
Oct 30 04:24:36.173420 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:24:36.173389    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c53e382_3a2d_438a_8093_aef7966c1913.slice/crio-211664cd1be8a78acec907065ddd7c90608f5f96e19a0fb8b47d96bc937c17d6 WatchSource:0}: Error finding container 211664cd1be8a78acec907065ddd7c90608f5f96e19a0fb8b47d96bc937c17d6: Status 404 returned error can't find the container with id 211664cd1be8a78acec907065ddd7c90608f5f96e19a0fb8b47d96bc937c17d6
Oct 30 04:24:36.637751 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:36.637718    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8" event={"ID":"9c53e382-3a2d-438a-8093-aef7966c1913","Type":"ContainerStarted","Data":"0a76a029931677415fc594d83b07e2fe2874c0e680241ed983a7c7ef9674111b"}
Oct 30 04:24:36.637927 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:36.637769    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8" event={"ID":"9c53e382-3a2d-438a-8093-aef7966c1913","Type":"ContainerStarted","Data":"211664cd1be8a78acec907065ddd7c90608f5f96e19a0fb8b47d96bc937c17d6"}
Oct 30 04:24:36.637927 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:36.637791    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8"
Oct 30 04:24:36.683980 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:36.683932    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8" podStartSLOduration=2.354724003 podStartE2EDuration="2.683916504s" totalImagesPullingTime="329.192501ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:24:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:24:36.681228446 +0000 UTC m=+1596.093624018" watchObservedRunningTime="2026-10-30 04:24:36.683916504 +0000 UTC m=+1596.096312079"
Oct 30 04:24:42.648700 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:42.648672    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8"
Oct 30 04:24:50.858153 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:50.858114    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:24:53.421909 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:53.421875    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l"]
Oct 30 04:24:53.425818 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:53.425796    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l"
Oct 30 04:24:53.444882 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:53.444852    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l"]
Oct 30 04:24:53.453079 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:53.453053    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-397bda\""
Oct 30 04:24:53.498440 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:53.498404    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltpqm\" (UniqueName: \"kubernetes.io/projected/2e928851-f639-4c63-bf99-3243a4f50edd-kube-api-access-ltpqm\") pod \"payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l\" (UID: \"2e928851-f639-4c63-bf99-3243a4f50edd\") " pod="openshift-ingress/payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l"
Oct 30 04:24:53.498618 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:53.498469    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e928851-f639-4c63-bf99-3243a4f50edd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l\" (UID: \"2e928851-f639-4c63-bf99-3243a4f50edd\") " pod="openshift-ingress/payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l"
Oct 30 04:24:53.599699 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:53.599650    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltpqm\" (UniqueName: \"kubernetes.io/projected/2e928851-f639-4c63-bf99-3243a4f50edd-kube-api-access-ltpqm\") pod \"payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l\" (UID: \"2e928851-f639-4c63-bf99-3243a4f50edd\") " pod="openshift-ingress/payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l"
Oct 30 04:24:53.610314 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:53.610267    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e928851-f639-4c63-bf99-3243a4f50edd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l\" (UID: \"2e928851-f639-4c63-bf99-3243a4f50edd\") " pod="openshift-ingress/payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l"
Oct 30 04:24:53.625372 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:53.625330    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e928851-f639-4c63-bf99-3243a4f50edd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l\" (UID: \"2e928851-f639-4c63-bf99-3243a4f50edd\") " pod="openshift-ingress/payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l"
Oct 30 04:24:53.685315 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:53.679276    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltpqm\" (UniqueName: \"kubernetes.io/projected/2e928851-f639-4c63-bf99-3243a4f50edd-kube-api-access-ltpqm\") pod \"payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l\" (UID: \"2e928851-f639-4c63-bf99-3243a4f50edd\") " pod="openshift-ingress/payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l"
Oct 30 04:24:53.739548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:53.739511    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l"
Oct 30 04:24:54.327077 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:54.327049    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l"]
Oct 30 04:24:54.328541 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:24:54.328506    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e928851_f639_4c63_bf99_3243a4f50edd.slice/crio-bba7e4dc6957457350676188dc5e34d1670bb16baaf14c8a632d09a219bbc417 WatchSource:0}: Error finding container bba7e4dc6957457350676188dc5e34d1670bb16baaf14c8a632d09a219bbc417: Status 404 returned error can't find the container with id bba7e4dc6957457350676188dc5e34d1670bb16baaf14c8a632d09a219bbc417
Oct 30 04:24:54.791605 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:54.791564    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l" event={"ID":"2e928851-f639-4c63-bf99-3243a4f50edd","Type":"ContainerStarted","Data":"f7430c704f554205c9b55663939282ba8dcf6ff311803cde00e1c4649729d50f"}
Oct 30 04:24:54.792024 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:54.791624    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l" event={"ID":"2e928851-f639-4c63-bf99-3243a4f50edd","Type":"ContainerStarted","Data":"bba7e4dc6957457350676188dc5e34d1670bb16baaf14c8a632d09a219bbc417"}
Oct 30 04:24:54.792024 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:54.791687    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l"
Oct 30 04:24:54.833449 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:24:54.833401    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l" podStartSLOduration=1.833385443 podStartE2EDuration="1.833385443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:24:54.83040039 +0000 UTC m=+1614.242795962" watchObservedRunningTime="2026-10-30 04:24:54.833385443 +0000 UTC m=+1614.245781025"
Oct 30 04:24:55.614066 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:24:55.614029    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:25:01.836056 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:01.836024    4874 generic.go:356] "Generic (PLEG): container finished" podID="32b4cf74-389e-4749-bf55-a1c52313e609" containerID="eb093c1b82de3a92d2c5265b0ef0d70c2ffe1e5073602d271324afc37aa33a5a" exitCode=0
Oct 30 04:25:01.836538 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:01.836064    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh" event={"ID":"32b4cf74-389e-4749-bf55-a1c52313e609","Type":"ContainerDied","Data":"eb093c1b82de3a92d2c5265b0ef0d70c2ffe1e5073602d271324afc37aa33a5a"}
Oct 30 04:25:01.836538 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:01.836475    4874 scope.go:122] "RemoveContainer" containerID="eb093c1b82de3a92d2c5265b0ef0d70c2ffe1e5073602d271324afc37aa33a5a"
Oct 30 04:25:01.837933 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:01.837903    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:25:01.837993 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:01.837971    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:25:01.838029 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:01.838003    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:25:02.843951 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:02.843909    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh" event={"ID":"32b4cf74-389e-4749-bf55-a1c52313e609","Type":"ContainerStarted","Data":"0bfbecdd7ea4eb2cb6aa00c75489eaf9987ae54867b4b61c7abcfbadec2d2f8b"}
Oct 30 04:25:03.501318 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:03.501285    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:25:03.524679 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:03.524640    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:25:03.526314 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:03.526275    4874 patch_prober.go:29] interesting pod/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 04:25:03.526440 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:03.526349    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh" podUID="32b4cf74-389e-4749-bf55-a1c52313e609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 04:25:04.525043 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:04.525005    4874 patch_prober.go:29] interesting pod/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 04:25:04.525428 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:04.525071    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh" podUID="32b4cf74-389e-4749-bf55-a1c52313e609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 04:25:05.524546 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:05.524514    4874 patch_prober.go:29] interesting pod/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 04:25:05.524741 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:05.524572    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh" podUID="32b4cf74-389e-4749-bf55-a1c52313e609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 04:25:05.800529 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:05.800447    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l"
Oct 30 04:25:06.525072 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:06.525027    4874 patch_prober.go:29] interesting pod/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 04:25:06.525465 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:06.525420    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh" podUID="32b4cf74-389e-4749-bf55-a1c52313e609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 04:25:07.042598 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:07.042560    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4"]
Oct 30 04:25:07.524858 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:07.524820    4874 patch_prober.go:29] interesting pod/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 04:25:07.525029 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:07.524882    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh" podUID="32b4cf74-389e-4749-bf55-a1c52313e609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 04:25:08.529023 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:08.528993    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:25:08.529933 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:08.529913    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:25:12.043927 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:12.043887    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4" podUID="feabd4f1-bcc9-4529-9001-4580967be5d1" containerName="payload-pre-processing" containerID="cri-o://968d2ef42fa2d6b2376facdfb0db64982b03da6cca2fab3ebcec854ae678613a" gracePeriod=25
Oct 30 04:25:16.033024 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:16.032985    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4" podUID="feabd4f1-bcc9-4529-9001-4580967be5d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:25:18.078571 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:18.078531    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:25:21.631423 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:21.631226    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch"]
Oct 30 04:25:21.631423 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:21.631480    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch" podUID="65809ef9-6fc6-4dd9-b224-0881ea5cad6c" containerName="maas-api" containerID="cri-o://3e4bb00bc4cc27d38c5ff9261c581dc4ec99cc959155be12707f11c9d21b532b" gracePeriod=30
Oct 30 04:25:21.997539 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:21.997513    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch"
Oct 30 04:25:22.112274 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:22.085681    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-openshift-service-ca\") pod \"65809ef9-6fc6-4dd9-b224-0881ea5cad6c\" (UID: \"65809ef9-6fc6-4dd9-b224-0881ea5cad6c\") "
Oct 30 04:25:22.112274 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:22.085760    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-metrics-certs\") pod \"65809ef9-6fc6-4dd9-b224-0881ea5cad6c\" (UID: \"65809ef9-6fc6-4dd9-b224-0881ea5cad6c\") "
Oct 30 04:25:22.112274 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:22.085832    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-kube-api-access-8blct\" (UniqueName: \"kubernetes.io/projected/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-kube-api-access-8blct\") pod \"65809ef9-6fc6-4dd9-b224-0881ea5cad6c\" (UID: \"65809ef9-6fc6-4dd9-b224-0881ea5cad6c\") "
Oct 30 04:25:22.112274 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:22.085983    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-maas-api-tls\") pod \"65809ef9-6fc6-4dd9-b224-0881ea5cad6c\" (UID: \"65809ef9-6fc6-4dd9-b224-0881ea5cad6c\") "
Oct 30 04:25:22.117459 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:22.117420    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-openshift-service-ca" pod "65809ef9-6fc6-4dd9-b224-0881ea5cad6c" (UID: "65809ef9-6fc6-4dd9-b224-0881ea5cad6c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:25:22.192290 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:22.187469    4874 generic.go:356] "Generic (PLEG): container finished" podID="65809ef9-6fc6-4dd9-b224-0881ea5cad6c" containerID="3e4bb00bc4cc27d38c5ff9261c581dc4ec99cc959155be12707f11c9d21b532b" exitCode=0
Oct 30 04:25:22.192290 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:22.187676    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch" event={"ID":"65809ef9-6fc6-4dd9-b224-0881ea5cad6c","Type":"ContainerDied","Data":"3e4bb00bc4cc27d38c5ff9261c581dc4ec99cc959155be12707f11c9d21b532b"}
Oct 30 04:25:22.192290 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:22.187775    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch" event={"ID":"65809ef9-6fc6-4dd9-b224-0881ea5cad6c","Type":"ContainerDied","Data":"5d392640679f9efaaa1aff3a5d356c51581ef862b68b9875d755ece13852a5d3"}
Oct 30 04:25:22.192290 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:22.187825    4874 scope.go:122] "RemoveContainer" containerID="3e4bb00bc4cc27d38c5ff9261c581dc4ec99cc959155be12707f11c9d21b532b"
Oct 30 04:25:22.192290 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:22.188019    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch"
Oct 30 04:25:22.192290 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:22.189393    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:22.210590 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:22.210488    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-metrics-certs" pod "65809ef9-6fc6-4dd9-b224-0881ea5cad6c" (UID: "65809ef9-6fc6-4dd9-b224-0881ea5cad6c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:25:22.217654 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:22.217618    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-kube-api-access-8blct" pod "65809ef9-6fc6-4dd9-b224-0881ea5cad6c" (UID: "65809ef9-6fc6-4dd9-b224-0881ea5cad6c"). InnerVolumeSpecName "kube-api-access-8blct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:25:22.233443 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:22.233409    4874 scope.go:122] "RemoveContainer" containerID="3e4bb00bc4cc27d38c5ff9261c581dc4ec99cc959155be12707f11c9d21b532b"
Oct 30 04:25:22.233865 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:25:22.233833    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e4bb00bc4cc27d38c5ff9261c581dc4ec99cc959155be12707f11c9d21b532b\": container with ID starting with 3e4bb00bc4cc27d38c5ff9261c581dc4ec99cc959155be12707f11c9d21b532b not found: ID does not exist" containerID="3e4bb00bc4cc27d38c5ff9261c581dc4ec99cc959155be12707f11c9d21b532b"
Oct 30 04:25:22.233991 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:22.233877    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4bb00bc4cc27d38c5ff9261c581dc4ec99cc959155be12707f11c9d21b532b"} err="failed to get container status \"3e4bb00bc4cc27d38c5ff9261c581dc4ec99cc959155be12707f11c9d21b532b\": rpc error: code = NotFound desc = could not find container \"3e4bb00bc4cc27d38c5ff9261c581dc4ec99cc959155be12707f11c9d21b532b\": container with ID starting with 3e4bb00bc4cc27d38c5ff9261c581dc4ec99cc959155be12707f11c9d21b532b not found: ID does not exist"
Oct 30 04:25:22.234462 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:22.234436    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-maas-api-tls" pod "65809ef9-6fc6-4dd9-b224-0881ea5cad6c" (UID: "65809ef9-6fc6-4dd9-b224-0881ea5cad6c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:25:22.290402 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:22.290376    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8blct\" (UniqueName: \"kubernetes.io/projected/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-kube-api-access-8blct\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:22.290402 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:22.290400    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:22.290533 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:22.290415    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65809ef9-6fc6-4dd9-b224-0881ea5cad6c-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:22.555298 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:22.555266    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch"]
Oct 30 04:25:22.558957 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:22.558931    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e124dc-597bc66f5d-kqpch"]
Oct 30 04:25:22.728842 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:22.728812    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65809ef9-6fc6-4dd9-b224-0881ea5cad6c" path="/var/lib/kubelet/pods/65809ef9-6fc6-4dd9-b224-0881ea5cad6c/volumes"
Oct 30 04:25:24.094007 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:24.093972    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"]
Oct 30 04:25:24.094471 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:24.094153    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65" podUID="16811f80-e367-4e31-b03c-82efdff4a3bb" containerName="istio-proxy" containerID="cri-o://6ff68326899b4d992d4da129b13d399c544a3a7c4927637743b138660800de49" gracePeriod=30
Oct 30 04:25:25.609466 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:25:25.609432    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:25:26.033305 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:26.033272    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4" podUID="feabd4f1-bcc9-4529-9001-4580967be5d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:25:28.222610 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:28.222574    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"]
Oct 30 04:25:28.223111 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:28.222829    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42" podUID="f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" containerName="istio-proxy" containerID="cri-o://1d8143bdcb14ed4d08d7da80b45e6f470bde7c97362d69d5235fe50ac5654eaf" gracePeriod=30
Oct 30 04:25:29.230750 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.230712    4874 generic.go:356] "Generic (PLEG): container finished" podID="16811f80-e367-4e31-b03c-82efdff4a3bb" containerID="6ff68326899b4d992d4da129b13d399c544a3a7c4927637743b138660800de49" exitCode=0
Oct 30 04:25:29.231063 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.230766    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65" event={"ID":"16811f80-e367-4e31-b03c-82efdff4a3bb","Type":"ContainerDied","Data":"6ff68326899b4d992d4da129b13d399c544a3a7c4927637743b138660800de49"}
Oct 30 04:25:29.349037 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.349011    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:25:29.510612 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.510582    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-envoy\") pod \"16811f80-e367-4e31-b03c-82efdff4a3bb\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") "
Oct 30 04:25:29.510813 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.510641    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-workload-certs\") pod \"16811f80-e367-4e31-b03c-82efdff4a3bb\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") "
Oct 30 04:25:29.510813 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.510669    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-token\" (UniqueName: \"kubernetes.io/projected/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-token\") pod \"16811f80-e367-4e31-b03c-82efdff4a3bb\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") "
Oct 30 04:25:29.510813 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.510707    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-podinfo\") pod \"16811f80-e367-4e31-b03c-82efdff4a3bb\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") "
Oct 30 04:25:29.510813 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.510809    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16811f80-e367-4e31-b03c-82efdff4a3bb-kube-api-access-l896t\" (UniqueName: \"kubernetes.io/projected/16811f80-e367-4e31-b03c-82efdff4a3bb-kube-api-access-l896t\") pod \"16811f80-e367-4e31-b03c-82efdff4a3bb\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") "
Oct 30 04:25:29.511022 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.510876    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-workload-socket\") pod \"16811f80-e367-4e31-b03c-82efdff4a3bb\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") "
Oct 30 04:25:29.511022 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.510915    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-credential-socket\") pod \"16811f80-e367-4e31-b03c-82efdff4a3bb\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") "
Oct 30 04:25:29.511022 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.510913    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-workload-certs" pod "16811f80-e367-4e31-b03c-82efdff4a3bb" (UID: "16811f80-e367-4e31-b03c-82efdff4a3bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:25:29.511022 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.510973    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-data\") pod \"16811f80-e367-4e31-b03c-82efdff4a3bb\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") "
Oct 30 04:25:29.511216 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.511032    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16811f80-e367-4e31-b03c-82efdff4a3bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16811f80-e367-4e31-b03c-82efdff4a3bb-istiod-ca-cert\") pod \"16811f80-e367-4e31-b03c-82efdff4a3bb\" (UID: \"16811f80-e367-4e31-b03c-82efdff4a3bb\") "
Oct 30 04:25:29.511525 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.511481    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-workload-socket" pod "16811f80-e367-4e31-b03c-82efdff4a3bb" (UID: "16811f80-e367-4e31-b03c-82efdff4a3bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:25:29.511984 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.511891    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16811f80-e367-4e31-b03c-82efdff4a3bb-istiod-ca-cert" pod "16811f80-e367-4e31-b03c-82efdff4a3bb" (UID: "16811f80-e367-4e31-b03c-82efdff4a3bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:25:29.512687 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.512665    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:29.512813 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.512800    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:29.512900 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.512888    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16811f80-e367-4e31-b03c-82efdff4a3bb-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:29.513365 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.513327    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-token" pod "16811f80-e367-4e31-b03c-82efdff4a3bb" (UID: "16811f80-e367-4e31-b03c-82efdff4a3bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:25:29.513697 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.513658    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-credential-socket" pod "16811f80-e367-4e31-b03c-82efdff4a3bb" (UID: "16811f80-e367-4e31-b03c-82efdff4a3bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:25:29.520725 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.520653    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-data" pod "16811f80-e367-4e31-b03c-82efdff4a3bb" (UID: "16811f80-e367-4e31-b03c-82efdff4a3bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:25:29.544813 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.544766    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-envoy" pod "16811f80-e367-4e31-b03c-82efdff4a3bb" (UID: "16811f80-e367-4e31-b03c-82efdff4a3bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:25:29.556440 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.556139    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-podinfo" pod "16811f80-e367-4e31-b03c-82efdff4a3bb" (UID: "16811f80-e367-4e31-b03c-82efdff4a3bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:25:29.556440 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.556274    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16811f80-e367-4e31-b03c-82efdff4a3bb-kube-api-access-l896t" pod "16811f80-e367-4e31-b03c-82efdff4a3bb" (UID: "16811f80-e367-4e31-b03c-82efdff4a3bb"). InnerVolumeSpecName "kube-api-access-l896t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:25:29.613947 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.613913    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:29.613947 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.613939    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:29.613947 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.613948    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:29.614170 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.613957    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l896t\" (UniqueName: \"kubernetes.io/projected/16811f80-e367-4e31-b03c-82efdff4a3bb-kube-api-access-l896t\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:29.614170 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.613968    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:29.614170 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.613977    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16811f80-e367-4e31-b03c-82efdff4a3bb-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:29.657871 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.657843    4874 patch_prober.go:29] interesting pod/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 04:25:29.658002 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:29.657918    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42" podUID="f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:25:30.237126 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:30.237098    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"
Oct 30 04:25:30.237126 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:30.237105    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65" event={"ID":"16811f80-e367-4e31-b03c-82efdff4a3bb","Type":"ContainerDied","Data":"a4eadbe208737834046f7d6187c9200ab7b71a24234f5828024c3de62ac3d99c"}
Oct 30 04:25:30.237630 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:30.237165    4874 scope.go:122] "RemoveContainer" containerID="6ff68326899b4d992d4da129b13d399c544a3a7c4927637743b138660800de49"
Oct 30 04:25:30.303871 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:30.303837    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"]
Oct 30 04:25:30.312777 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:30.312751    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-nfw65"]
Oct 30 04:25:30.729360 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:30.729330    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16811f80-e367-4e31-b03c-82efdff4a3bb" path="/var/lib/kubelet/pods/16811f80-e367-4e31-b03c-82efdff4a3bb/volumes"
Oct 30 04:25:32.267552 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:32.267464    4874 generic.go:356] "Generic (PLEG): container finished" podID="feabd4f1-bcc9-4529-9001-4580967be5d1" containerID="968d2ef42fa2d6b2376facdfb0db64982b03da6cca2fab3ebcec854ae678613a" exitCode=0
Oct 30 04:25:32.267552 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:32.267540    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4" event={"ID":"feabd4f1-bcc9-4529-9001-4580967be5d1","Type":"ContainerDied","Data":"968d2ef42fa2d6b2376facdfb0db64982b03da6cca2fab3ebcec854ae678613a"}
Oct 30 04:25:32.324657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:32.324631    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4"
Oct 30 04:25:32.358747 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:32.358719    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feabd4f1-bcc9-4529-9001-4580967be5d1-kube-api-access-w5xck\" (UniqueName: \"kubernetes.io/projected/feabd4f1-bcc9-4529-9001-4580967be5d1-kube-api-access-w5xck\") pod \"feabd4f1-bcc9-4529-9001-4580967be5d1\" (UID: \"feabd4f1-bcc9-4529-9001-4580967be5d1\") "
Oct 30 04:25:32.358874 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:32.358756    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feabd4f1-bcc9-4529-9001-4580967be5d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feabd4f1-bcc9-4529-9001-4580967be5d1-plugins-config-volume\") pod \"feabd4f1-bcc9-4529-9001-4580967be5d1\" (UID: \"feabd4f1-bcc9-4529-9001-4580967be5d1\") "
Oct 30 04:25:32.359218 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:32.359192    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feabd4f1-bcc9-4529-9001-4580967be5d1-plugins-config-volume" pod "feabd4f1-bcc9-4529-9001-4580967be5d1" (UID: "feabd4f1-bcc9-4529-9001-4580967be5d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:25:32.360845 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:32.360815    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feabd4f1-bcc9-4529-9001-4580967be5d1-kube-api-access-w5xck" pod "feabd4f1-bcc9-4529-9001-4580967be5d1" (UID: "feabd4f1-bcc9-4529-9001-4580967be5d1"). InnerVolumeSpecName "kube-api-access-w5xck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:25:32.459906 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:32.459862    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5xck\" (UniqueName: \"kubernetes.io/projected/feabd4f1-bcc9-4529-9001-4580967be5d1-kube-api-access-w5xck\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:32.459906 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:32.459901    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feabd4f1-bcc9-4529-9001-4580967be5d1-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:33.274825 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.274802    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4"
Oct 30 04:25:33.275213 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.274833    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4" event={"ID":"feabd4f1-bcc9-4529-9001-4580967be5d1","Type":"ContainerDied","Data":"098b41f5d3b342dcc8dcc9093005b2f3a7d8ec9fbb642c61c52d9de8d952823f"}
Oct 30 04:25:33.275213 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.274890    4874 scope.go:122] "RemoveContainer" containerID="968d2ef42fa2d6b2376facdfb0db64982b03da6cca2fab3ebcec854ae678613a"
Oct 30 04:25:33.351669 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.351642    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4"]
Oct 30 04:25:33.360505 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.360480    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-9pbd4"]
Oct 30 04:25:33.487825 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.487803    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:25:33.574638 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.574566    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-token\" (UniqueName: \"kubernetes.io/projected/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-token\") pod \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") "
Oct 30 04:25:33.574780 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.574642    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-data\") pod \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") "
Oct 30 04:25:33.574780 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.574680    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-kube-api-access-5h7vc\" (UniqueName: \"kubernetes.io/projected/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-kube-api-access-5h7vc\") pod \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") "
Oct 30 04:25:33.574780 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.574710    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istiod-ca-cert\") pod \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") "
Oct 30 04:25:33.574780 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.574755    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-envoy\") pod \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") "
Oct 30 04:25:33.575011 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.574890    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-credential-socket\") pod \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") "
Oct 30 04:25:33.575011 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.574960    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-workload-socket\") pod \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") "
Oct 30 04:25:33.575128 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.575009    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-podinfo\") pod \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") "
Oct 30 04:25:33.575128 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.575096    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-workload-certs\") pod \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\" (UID: \"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d\") "
Oct 30 04:25:33.575225 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.575005    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-data" pod "f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" (UID: "f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:25:33.575323 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.575194    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-workload-socket" pod "f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" (UID: "f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:25:33.575475 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.575452    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-credential-socket" pod "f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" (UID: "f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:25:33.575536 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.575494    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-workload-certs" pod "f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" (UID: "f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:25:33.575631 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.575606    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istiod-ca-cert" pod "f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" (UID: "f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:25:33.575758 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.575717    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:33.575758 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.575737    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:33.575758 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.575752    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:33.575927 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.575765    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:33.575927 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.575778    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:33.577070 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.577048    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-token" pod "f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" (UID: "f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:25:33.577324 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.577307    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-kube-api-access-5h7vc" pod "f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" (UID: "f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d"). InnerVolumeSpecName "kube-api-access-5h7vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:25:33.577457 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.577438    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-podinfo" pod "f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" (UID: "f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:25:33.577788 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.577773    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-envoy" pod "f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" (UID: "f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:25:33.676939 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.676915    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:33.676939 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.676938    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h7vc\" (UniqueName: \"kubernetes.io/projected/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-kube-api-access-5h7vc\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:33.677060 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.676946    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:33.677060 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:33.676954    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:34.283479 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:34.283443    4874 generic.go:356] "Generic (PLEG): container finished" podID="f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" containerID="1d8143bdcb14ed4d08d7da80b45e6f470bde7c97362d69d5235fe50ac5654eaf" exitCode=0
Oct 30 04:25:34.283959 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:34.283507    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"
Oct 30 04:25:34.283959 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:34.283515    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42" event={"ID":"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d","Type":"ContainerDied","Data":"1d8143bdcb14ed4d08d7da80b45e6f470bde7c97362d69d5235fe50ac5654eaf"}
Oct 30 04:25:34.283959 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:34.283597    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42" event={"ID":"f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d","Type":"ContainerDied","Data":"e648dad55fd852f3200e23a4eaf30f0961d9232a21d77896d975088be8ff4c68"}
Oct 30 04:25:34.283959 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:34.283621    4874 scope.go:122] "RemoveContainer" containerID="1d8143bdcb14ed4d08d7da80b45e6f470bde7c97362d69d5235fe50ac5654eaf"
Oct 30 04:25:34.293866 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:34.293841    4874 scope.go:122] "RemoveContainer" containerID="ad18e184286cb701f009df217f4d7d3be4b49a6b04579cc50ed46f720da83e1b"
Oct 30 04:25:34.302376 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:34.302361    4874 scope.go:122] "RemoveContainer" containerID="1d8143bdcb14ed4d08d7da80b45e6f470bde7c97362d69d5235fe50ac5654eaf"
Oct 30 04:25:34.302642 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:25:34.302621    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8143bdcb14ed4d08d7da80b45e6f470bde7c97362d69d5235fe50ac5654eaf\": container with ID starting with 1d8143bdcb14ed4d08d7da80b45e6f470bde7c97362d69d5235fe50ac5654eaf not found: ID does not exist" containerID="1d8143bdcb14ed4d08d7da80b45e6f470bde7c97362d69d5235fe50ac5654eaf"
Oct 30 04:25:34.302732 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:34.302648    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8143bdcb14ed4d08d7da80b45e6f470bde7c97362d69d5235fe50ac5654eaf"} err="failed to get container status \"1d8143bdcb14ed4d08d7da80b45e6f470bde7c97362d69d5235fe50ac5654eaf\": rpc error: code = NotFound desc = could not find container \"1d8143bdcb14ed4d08d7da80b45e6f470bde7c97362d69d5235fe50ac5654eaf\": container with ID starting with 1d8143bdcb14ed4d08d7da80b45e6f470bde7c97362d69d5235fe50ac5654eaf not found: ID does not exist"
Oct 30 04:25:34.302732 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:34.302665    4874 scope.go:122] "RemoveContainer" containerID="ad18e184286cb701f009df217f4d7d3be4b49a6b04579cc50ed46f720da83e1b"
Oct 30 04:25:34.302934 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:25:34.302912    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad18e184286cb701f009df217f4d7d3be4b49a6b04579cc50ed46f720da83e1b\": container with ID starting with ad18e184286cb701f009df217f4d7d3be4b49a6b04579cc50ed46f720da83e1b not found: ID does not exist" containerID="ad18e184286cb701f009df217f4d7d3be4b49a6b04579cc50ed46f720da83e1b"
Oct 30 04:25:34.302976 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:34.302944    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad18e184286cb701f009df217f4d7d3be4b49a6b04579cc50ed46f720da83e1b"} err="failed to get container status \"ad18e184286cb701f009df217f4d7d3be4b49a6b04579cc50ed46f720da83e1b\": rpc error: code = NotFound desc = could not find container \"ad18e184286cb701f009df217f4d7d3be4b49a6b04579cc50ed46f720da83e1b\": container with ID starting with ad18e184286cb701f009df217f4d7d3be4b49a6b04579cc50ed46f720da83e1b not found: ID does not exist"
Oct 30 04:25:34.345397 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:34.345374    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"]
Oct 30 04:25:34.357049 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:34.357027    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e124dc-openshift-default-7b8dccd995-4xk42"]
Oct 30 04:25:34.734400 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:34.734371    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" path="/var/lib/kubelet/pods/f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d/volumes"
Oct 30 04:25:34.734787 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:34.734773    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feabd4f1-bcc9-4529-9001-4580967be5d1" path="/var/lib/kubelet/pods/feabd4f1-bcc9-4529-9001-4580967be5d1/volumes"
Oct 30 04:25:36.609267 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:36.599578    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"]
Oct 30 04:25:36.609267 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:36.599855    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj" podUID="10b28e83-cdc5-4591-858b-80df98885d4e" containerName="istio-proxy" containerID="cri-o://2bb175b2ac9439e18009ac4789561f0ab4f8f4196252ee5ce48c9e0fdc5cfff2" gracePeriod=30
Oct 30 04:25:39.177019 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:39.176986    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:25:40.191604 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:40.191565    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt"]
Oct 30 04:25:42.176560 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.176478    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:25:42.362605 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.362501    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-istio-data\") pod \"10b28e83-cdc5-4591-858b-80df98885d4e\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") "
Oct 30 04:25:42.362605 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.362555    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-workload-socket\") pod \"10b28e83-cdc5-4591-858b-80df98885d4e\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") "
Oct 30 04:25:42.362605 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.362597    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-istio-envoy\") pod \"10b28e83-cdc5-4591-858b-80df98885d4e\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") "
Oct 30 04:25:42.362919 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.362661    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-workload-certs\") pod \"10b28e83-cdc5-4591-858b-80df98885d4e\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") "
Oct 30 04:25:42.362919 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.362738    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-credential-socket\") pod \"10b28e83-cdc5-4591-858b-80df98885d4e\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") "
Oct 30 04:25:42.362919 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.362797    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b28e83-cdc5-4591-858b-80df98885d4e-kube-api-access-mjpd7\" (UniqueName: \"kubernetes.io/projected/10b28e83-cdc5-4591-858b-80df98885d4e-kube-api-access-mjpd7\") pod \"10b28e83-cdc5-4591-858b-80df98885d4e\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") "
Oct 30 04:25:42.362919 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.362810    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-istio-data" pod "10b28e83-cdc5-4591-858b-80df98885d4e" (UID: "10b28e83-cdc5-4591-858b-80df98885d4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:25:42.362919 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.362862    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10b28e83-cdc5-4591-858b-80df98885d4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b28e83-cdc5-4591-858b-80df98885d4e-istio-podinfo\") pod \"10b28e83-cdc5-4591-858b-80df98885d4e\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") "
Oct 30 04:25:42.362919 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.362906    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b28e83-cdc5-4591-858b-80df98885d4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b28e83-cdc5-4591-858b-80df98885d4e-istiod-ca-cert\") pod \"10b28e83-cdc5-4591-858b-80df98885d4e\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") "
Oct 30 04:25:42.363220 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.362949    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b28e83-cdc5-4591-858b-80df98885d4e-istio-token\" (UniqueName: \"kubernetes.io/projected/10b28e83-cdc5-4591-858b-80df98885d4e-istio-token\") pod \"10b28e83-cdc5-4591-858b-80df98885d4e\" (UID: \"10b28e83-cdc5-4591-858b-80df98885d4e\") "
Oct 30 04:25:42.363578 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.363451    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:42.363695 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.363599    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-workload-certs" pod "10b28e83-cdc5-4591-858b-80df98885d4e" (UID: "10b28e83-cdc5-4591-858b-80df98885d4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:25:42.364175 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.364122    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-credential-socket" pod "10b28e83-cdc5-4591-858b-80df98885d4e" (UID: "10b28e83-cdc5-4591-858b-80df98885d4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:25:42.364790 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.364746    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b28e83-cdc5-4591-858b-80df98885d4e-istiod-ca-cert" pod "10b28e83-cdc5-4591-858b-80df98885d4e" (UID: "10b28e83-cdc5-4591-858b-80df98885d4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:25:42.367222 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.366550    4874 generic.go:356] "Generic (PLEG): container finished" podID="10b28e83-cdc5-4591-858b-80df98885d4e" containerID="2bb175b2ac9439e18009ac4789561f0ab4f8f4196252ee5ce48c9e0fdc5cfff2" exitCode=0
Oct 30 04:25:42.367222 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.366632    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj" event={"ID":"10b28e83-cdc5-4591-858b-80df98885d4e","Type":"ContainerDied","Data":"2bb175b2ac9439e18009ac4789561f0ab4f8f4196252ee5ce48c9e0fdc5cfff2"}
Oct 30 04:25:42.367222 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.366684    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj" event={"ID":"10b28e83-cdc5-4591-858b-80df98885d4e","Type":"ContainerDied","Data":"ad73c7d3a82c333a7fec58bfed95f090878db2f692b8514459abc2e9d650b57a"}
Oct 30 04:25:42.367222 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.366716    4874 scope.go:122] "RemoveContainer" containerID="2bb175b2ac9439e18009ac4789561f0ab4f8f4196252ee5ce48c9e0fdc5cfff2"
Oct 30 04:25:42.367222 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.366864    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"
Oct 30 04:25:42.401227 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.399724    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-workload-socket" pod "10b28e83-cdc5-4591-858b-80df98885d4e" (UID: "10b28e83-cdc5-4591-858b-80df98885d4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:25:42.447650 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.420404    4874 scope.go:122] "RemoveContainer" containerID="a3938d264c0c2e896b85ff65153cf0422811442940c3ab85b93c372e65b1cd62"
Oct 30 04:25:42.447650 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.425428    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b28e83-cdc5-4591-858b-80df98885d4e-kube-api-access-mjpd7" pod "10b28e83-cdc5-4591-858b-80df98885d4e" (UID: "10b28e83-cdc5-4591-858b-80df98885d4e"). InnerVolumeSpecName "kube-api-access-mjpd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:25:42.449463 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.449411    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b28e83-cdc5-4591-858b-80df98885d4e-istio-token" pod "10b28e83-cdc5-4591-858b-80df98885d4e" (UID: "10b28e83-cdc5-4591-858b-80df98885d4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:25:42.451792 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.451356    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-istio-envoy" pod "10b28e83-cdc5-4591-858b-80df98885d4e" (UID: "10b28e83-cdc5-4591-858b-80df98885d4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:25:42.451792 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.451435    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10b28e83-cdc5-4591-858b-80df98885d4e-istio-podinfo" pod "10b28e83-cdc5-4591-858b-80df98885d4e" (UID: "10b28e83-cdc5-4591-858b-80df98885d4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:25:42.499691 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.471128    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:42.499691 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.471161    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:42.499691 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.471177    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:42.499691 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.471190    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b28e83-cdc5-4591-858b-80df98885d4e-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:42.499691 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.471205    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjpd7\" (UniqueName: \"kubernetes.io/projected/10b28e83-cdc5-4591-858b-80df98885d4e-kube-api-access-mjpd7\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:42.499691 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.471219    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b28e83-cdc5-4591-858b-80df98885d4e-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:42.499691 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.471233    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b28e83-cdc5-4591-858b-80df98885d4e-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:42.499691 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.471264    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b28e83-cdc5-4591-858b-80df98885d4e-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:42.537302 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.525513    4874 scope.go:122] "RemoveContainer" containerID="2bb175b2ac9439e18009ac4789561f0ab4f8f4196252ee5ce48c9e0fdc5cfff2"
Oct 30 04:25:42.594305 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:25:42.591417    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bb175b2ac9439e18009ac4789561f0ab4f8f4196252ee5ce48c9e0fdc5cfff2\": container with ID starting with 2bb175b2ac9439e18009ac4789561f0ab4f8f4196252ee5ce48c9e0fdc5cfff2 not found: ID does not exist" containerID="2bb175b2ac9439e18009ac4789561f0ab4f8f4196252ee5ce48c9e0fdc5cfff2"
Oct 30 04:25:42.594305 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.591476    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bb175b2ac9439e18009ac4789561f0ab4f8f4196252ee5ce48c9e0fdc5cfff2"} err="failed to get container status \"2bb175b2ac9439e18009ac4789561f0ab4f8f4196252ee5ce48c9e0fdc5cfff2\": rpc error: code = NotFound desc = could not find container \"2bb175b2ac9439e18009ac4789561f0ab4f8f4196252ee5ce48c9e0fdc5cfff2\": container with ID starting with 2bb175b2ac9439e18009ac4789561f0ab4f8f4196252ee5ce48c9e0fdc5cfff2 not found: ID does not exist"
Oct 30 04:25:42.594305 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.591506    4874 scope.go:122] "RemoveContainer" containerID="a3938d264c0c2e896b85ff65153cf0422811442940c3ab85b93c372e65b1cd62"
Oct 30 04:25:42.600073 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:25:42.599675    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3938d264c0c2e896b85ff65153cf0422811442940c3ab85b93c372e65b1cd62\": container with ID starting with a3938d264c0c2e896b85ff65153cf0422811442940c3ab85b93c372e65b1cd62 not found: ID does not exist" containerID="a3938d264c0c2e896b85ff65153cf0422811442940c3ab85b93c372e65b1cd62"
Oct 30 04:25:42.600073 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.599724    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3938d264c0c2e896b85ff65153cf0422811442940c3ab85b93c372e65b1cd62"} err="failed to get container status \"a3938d264c0c2e896b85ff65153cf0422811442940c3ab85b93c372e65b1cd62\": rpc error: code = NotFound desc = could not find container \"a3938d264c0c2e896b85ff65153cf0422811442940c3ab85b93c372e65b1cd62\": container with ID starting with a3938d264c0c2e896b85ff65153cf0422811442940c3ab85b93c372e65b1cd62 not found: ID does not exist"
Oct 30 04:25:42.861611 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.861546    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"]
Oct 30 04:25:42.875259 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:42.865228    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-8h4wj"]
Oct 30 04:25:44.313621 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:44.313544    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8"]
Oct 30 04:25:44.314053 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:44.313767    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8" podUID="9c53e382-3a2d-438a-8093-aef7966c1913" containerName="maas-api" containerID="cri-o://0a76a029931677415fc594d83b07e2fe2874c0e680241ed983a7c7ef9674111b" gracePeriod=30
Oct 30 04:25:44.578122 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:44.578066    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8"
Oct 30 04:25:44.679596 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:44.679562    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c53e382-3a2d-438a-8093-aef7966c1913-kube-api-access-2qnd9\" (UniqueName: \"kubernetes.io/projected/9c53e382-3a2d-438a-8093-aef7966c1913-kube-api-access-2qnd9\") pod \"9c53e382-3a2d-438a-8093-aef7966c1913\" (UID: \"9c53e382-3a2d-438a-8093-aef7966c1913\") "
Oct 30 04:25:44.679789 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:44.679629    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c53e382-3a2d-438a-8093-aef7966c1913-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c53e382-3a2d-438a-8093-aef7966c1913-metrics-certs\") pod \"9c53e382-3a2d-438a-8093-aef7966c1913\" (UID: \"9c53e382-3a2d-438a-8093-aef7966c1913\") "
Oct 30 04:25:44.679789 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:44.679722    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c53e382-3a2d-438a-8093-aef7966c1913-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c53e382-3a2d-438a-8093-aef7966c1913-maas-api-tls\") pod \"9c53e382-3a2d-438a-8093-aef7966c1913\" (UID: \"9c53e382-3a2d-438a-8093-aef7966c1913\") "
Oct 30 04:25:44.679912 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:44.679862    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c53e382-3a2d-438a-8093-aef7966c1913-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c53e382-3a2d-438a-8093-aef7966c1913-openshift-service-ca\") pod \"9c53e382-3a2d-438a-8093-aef7966c1913\" (UID: \"9c53e382-3a2d-438a-8093-aef7966c1913\") "
Oct 30 04:25:44.680317 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:44.680289    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c53e382-3a2d-438a-8093-aef7966c1913-openshift-service-ca" pod "9c53e382-3a2d-438a-8093-aef7966c1913" (UID: "9c53e382-3a2d-438a-8093-aef7966c1913"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:25:44.681771 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:44.681746    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c53e382-3a2d-438a-8093-aef7966c1913-metrics-certs" pod "9c53e382-3a2d-438a-8093-aef7966c1913" (UID: "9c53e382-3a2d-438a-8093-aef7966c1913"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:25:44.681862 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:44.681770    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c53e382-3a2d-438a-8093-aef7966c1913-kube-api-access-2qnd9" pod "9c53e382-3a2d-438a-8093-aef7966c1913" (UID: "9c53e382-3a2d-438a-8093-aef7966c1913"). InnerVolumeSpecName "kube-api-access-2qnd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:25:44.682181 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:44.682159    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c53e382-3a2d-438a-8093-aef7966c1913-maas-api-tls" pod "9c53e382-3a2d-438a-8093-aef7966c1913" (UID: "9c53e382-3a2d-438a-8093-aef7966c1913"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:25:44.764278 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:44.756174    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10b28e83-cdc5-4591-858b-80df98885d4e" path="/var/lib/kubelet/pods/10b28e83-cdc5-4591-858b-80df98885d4e/volumes"
Oct 30 04:25:44.783123 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:44.783094    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qnd9\" (UniqueName: \"kubernetes.io/projected/9c53e382-3a2d-438a-8093-aef7966c1913-kube-api-access-2qnd9\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:44.783287 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:44.783128    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c53e382-3a2d-438a-8093-aef7966c1913-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:44.783287 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:44.783143    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c53e382-3a2d-438a-8093-aef7966c1913-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:44.783287 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:44.783155    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c53e382-3a2d-438a-8093-aef7966c1913-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:45.212567 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:45.212323    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt" podUID="75d36063-1391-428f-955b-5a6d6f4cba1b" containerName="payload-processing" containerID="cri-o://346ae8f05a18bf6e72fec6289946b4afb66b738582c4958df378c06bad7997fb" gracePeriod=25
Oct 30 04:25:45.392629 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:45.392597    4874 generic.go:356] "Generic (PLEG): container finished" podID="75d36063-1391-428f-955b-5a6d6f4cba1b" containerID="346ae8f05a18bf6e72fec6289946b4afb66b738582c4958df378c06bad7997fb" exitCode=0
Oct 30 04:25:45.393188 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:45.392704    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt" event={"ID":"75d36063-1391-428f-955b-5a6d6f4cba1b","Type":"ContainerDied","Data":"346ae8f05a18bf6e72fec6289946b4afb66b738582c4958df378c06bad7997fb"}
Oct 30 04:25:45.395546 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:45.395520    4874 generic.go:356] "Generic (PLEG): container finished" podID="9c53e382-3a2d-438a-8093-aef7966c1913" containerID="0a76a029931677415fc594d83b07e2fe2874c0e680241ed983a7c7ef9674111b" exitCode=0
Oct 30 04:25:45.395664 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:45.395572    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8"
Oct 30 04:25:45.395664 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:45.395580    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8" event={"ID":"9c53e382-3a2d-438a-8093-aef7966c1913","Type":"ContainerDied","Data":"0a76a029931677415fc594d83b07e2fe2874c0e680241ed983a7c7ef9674111b"}
Oct 30 04:25:45.395664 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:45.395637    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8" event={"ID":"9c53e382-3a2d-438a-8093-aef7966c1913","Type":"ContainerDied","Data":"211664cd1be8a78acec907065ddd7c90608f5f96e19a0fb8b47d96bc937c17d6"}
Oct 30 04:25:45.395823 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:45.395673    4874 scope.go:122] "RemoveContainer" containerID="0a76a029931677415fc594d83b07e2fe2874c0e680241ed983a7c7ef9674111b"
Oct 30 04:25:45.409398 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:45.409370    4874 scope.go:122] "RemoveContainer" containerID="0a76a029931677415fc594d83b07e2fe2874c0e680241ed983a7c7ef9674111b"
Oct 30 04:25:45.409693 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:25:45.409664    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a76a029931677415fc594d83b07e2fe2874c0e680241ed983a7c7ef9674111b\": container with ID starting with 0a76a029931677415fc594d83b07e2fe2874c0e680241ed983a7c7ef9674111b not found: ID does not exist" containerID="0a76a029931677415fc594d83b07e2fe2874c0e680241ed983a7c7ef9674111b"
Oct 30 04:25:45.409813 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:45.409702    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a76a029931677415fc594d83b07e2fe2874c0e680241ed983a7c7ef9674111b"} err="failed to get container status \"0a76a029931677415fc594d83b07e2fe2874c0e680241ed983a7c7ef9674111b\": rpc error: code = NotFound desc = could not find container \"0a76a029931677415fc594d83b07e2fe2874c0e680241ed983a7c7ef9674111b\": container with ID starting with 0a76a029931677415fc594d83b07e2fe2874c0e680241ed983a7c7ef9674111b not found: ID does not exist"
Oct 30 04:25:45.451909 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:45.451879    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8"]
Oct 30 04:25:45.458596 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:45.458559    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0c530e-544f887fb6-4vvs8"]
Oct 30 04:25:45.519795 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:45.519767    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt"
Oct 30 04:25:45.599695 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:45.599666    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75d36063-1391-428f-955b-5a6d6f4cba1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d36063-1391-428f-955b-5a6d6f4cba1b-plugins-config-volume\") pod \"75d36063-1391-428f-955b-5a6d6f4cba1b\" (UID: \"75d36063-1391-428f-955b-5a6d6f4cba1b\") "
Oct 30 04:25:45.599844 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:45.599806    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75d36063-1391-428f-955b-5a6d6f4cba1b-kube-api-access-jwzw9\" (UniqueName: \"kubernetes.io/projected/75d36063-1391-428f-955b-5a6d6f4cba1b-kube-api-access-jwzw9\") pod \"75d36063-1391-428f-955b-5a6d6f4cba1b\" (UID: \"75d36063-1391-428f-955b-5a6d6f4cba1b\") "
Oct 30 04:25:45.600123 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:45.600087    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d36063-1391-428f-955b-5a6d6f4cba1b-plugins-config-volume" pod "75d36063-1391-428f-955b-5a6d6f4cba1b" (UID: "75d36063-1391-428f-955b-5a6d6f4cba1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:25:45.600417 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:45.600398    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d36063-1391-428f-955b-5a6d6f4cba1b-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:45.602201 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:45.602174    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d36063-1391-428f-955b-5a6d6f4cba1b-kube-api-access-jwzw9" pod "75d36063-1391-428f-955b-5a6d6f4cba1b" (UID: "75d36063-1391-428f-955b-5a6d6f4cba1b"). InnerVolumeSpecName "kube-api-access-jwzw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:25:45.702184 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:45.702150    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwzw9\" (UniqueName: \"kubernetes.io/projected/75d36063-1391-428f-955b-5a6d6f4cba1b-kube-api-access-jwzw9\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:46.402350 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:46.402318    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt"
Oct 30 04:25:46.402809 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:46.402327    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt" event={"ID":"75d36063-1391-428f-955b-5a6d6f4cba1b","Type":"ContainerDied","Data":"4c714222ba3554bfdd4b5813e1ed800d613b5af595d6e09b63c898d1ef85edb1"}
Oct 30 04:25:46.402809 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:46.402449    4874 scope.go:122] "RemoveContainer" containerID="346ae8f05a18bf6e72fec6289946b4afb66b738582c4958df378c06bad7997fb"
Oct 30 04:25:46.439528 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:46.439493    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt" podUID="75d36063-1391-428f-955b-5a6d6f4cba1b" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.67:50052\" within 1s: context deadline exceeded"
Oct 30 04:25:46.439630 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:25:46.439552    4874 logging.go:55] [core] [Channel #536 SubChannel #537]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.67:50052", ServerName: "10.134.0.67:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.67:50052: operation was canceled"
Oct 30 04:25:46.453122 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:46.453096    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt"]
Oct 30 04:25:46.464396 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:46.464370    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e3c43b-845d85c686-6hjjt"]
Oct 30 04:25:46.735013 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:46.734983    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75d36063-1391-428f-955b-5a6d6f4cba1b" path="/var/lib/kubelet/pods/75d36063-1391-428f-955b-5a6d6f4cba1b/volumes"
Oct 30 04:25:46.735397 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:46.735384    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c53e382-3a2d-438a-8093-aef7966c1913" path="/var/lib/kubelet/pods/9c53e382-3a2d-438a-8093-aef7966c1913/volumes"
Oct 30 04:25:49.394555 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:49.394526    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"]
Oct 30 04:25:49.394937 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:49.394786    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh" podUID="32b4cf74-389e-4749-bf55-a1c52313e609" containerName="istio-proxy" containerID="cri-o://0bfbecdd7ea4eb2cb6aa00c75489eaf9987ae54867b4b61c7abcfbadec2d2f8b" gracePeriod=30
Oct 30 04:25:50.494841 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.494806    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"]
Oct 30 04:25:50.495431 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495352    4874 cpu_manager.go:414] "containerMap: removing container" podUID="9c53e382-3a2d-438a-8093-aef7966c1913" containerName="maas-api"
Oct 30 04:25:50.495431 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495367    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c53e382-3a2d-438a-8093-aef7966c1913" containerName="maas-api"
Oct 30 04:25:50.495431 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495389    4874 cpu_manager.go:414] "containerMap: removing container" podUID="f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" containerName="istio-proxy"
Oct 30 04:25:50.495431 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495394    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" containerName="istio-proxy"
Oct 30 04:25:50.495431 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495402    4874 cpu_manager.go:414] "containerMap: removing container" podUID="f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" containerName="istio-proxy"
Oct 30 04:25:50.495431 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495407    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" containerName="istio-proxy"
Oct 30 04:25:50.495431 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495420    4874 cpu_manager.go:414] "containerMap: removing container" podUID="feabd4f1-bcc9-4529-9001-4580967be5d1" containerName="payload-pre-processing"
Oct 30 04:25:50.495431 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495425    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feabd4f1-bcc9-4529-9001-4580967be5d1" containerName="payload-pre-processing"
Oct 30 04:25:50.495431 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495436    4874 cpu_manager.go:414] "containerMap: removing container" podUID="10b28e83-cdc5-4591-858b-80df98885d4e" containerName="istio-proxy"
Oct 30 04:25:50.495703 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495440    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b28e83-cdc5-4591-858b-80df98885d4e" containerName="istio-proxy"
Oct 30 04:25:50.495703 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495452    4874 cpu_manager.go:414] "containerMap: removing container" podUID="10b28e83-cdc5-4591-858b-80df98885d4e" containerName="istio-proxy"
Oct 30 04:25:50.495703 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495457    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b28e83-cdc5-4591-858b-80df98885d4e" containerName="istio-proxy"
Oct 30 04:25:50.495703 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495464    4874 cpu_manager.go:414] "containerMap: removing container" podUID="16811f80-e367-4e31-b03c-82efdff4a3bb" containerName="istio-proxy"
Oct 30 04:25:50.495703 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495468    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16811f80-e367-4e31-b03c-82efdff4a3bb" containerName="istio-proxy"
Oct 30 04:25:50.495703 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495474    4874 cpu_manager.go:414] "containerMap: removing container" podUID="65809ef9-6fc6-4dd9-b224-0881ea5cad6c" containerName="maas-api"
Oct 30 04:25:50.495703 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495478    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65809ef9-6fc6-4dd9-b224-0881ea5cad6c" containerName="maas-api"
Oct 30 04:25:50.495703 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495487    4874 cpu_manager.go:414] "containerMap: removing container" podUID="75d36063-1391-428f-955b-5a6d6f4cba1b" containerName="payload-processing"
Oct 30 04:25:50.495703 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495491    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d36063-1391-428f-955b-5a6d6f4cba1b" containerName="payload-processing"
Oct 30 04:25:50.495703 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495558    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b28e83-cdc5-4591-858b-80df98885d4e" containerName="istio-proxy"
Oct 30 04:25:50.495703 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495566    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="16811f80-e367-4e31-b03c-82efdff4a3bb" containerName="istio-proxy"
Oct 30 04:25:50.495703 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495572    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="65809ef9-6fc6-4dd9-b224-0881ea5cad6c" containerName="maas-api"
Oct 30 04:25:50.495703 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495577    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d36063-1391-428f-955b-5a6d6f4cba1b" containerName="payload-processing"
Oct 30 04:25:50.495703 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495583    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c53e382-3a2d-438a-8093-aef7966c1913" containerName="maas-api"
Oct 30 04:25:50.495703 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495590    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" containerName="istio-proxy"
Oct 30 04:25:50.495703 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495594    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e8c8b4-5f08-4100-bc8b-ed0cabb94a3d" containerName="istio-proxy"
Oct 30 04:25:50.495703 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.495601    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="feabd4f1-bcc9-4529-9001-4580967be5d1" containerName="payload-pre-processing"
Oct 30 04:25:50.500529 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.500511    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.524447 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.524422    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"]
Oct 30 04:25:50.591263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.577010    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.591263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.577130    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.591263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.577212    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.591263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.577265    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.591263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.577364    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.591263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.577401    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.591263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.577435    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.591263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.577460    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.591263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.577484    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dvcw\" (UniqueName: \"kubernetes.io/projected/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-kube-api-access-9dvcw\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.678972 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.678936    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.679176 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.679004    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.679176 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.679051    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.679176 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.679081    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.679176 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.679160    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.679433 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.679193    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.679433 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.679226    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.679433 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.679276    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.679433 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.679303    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dvcw\" (UniqueName: \"kubernetes.io/projected/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-kube-api-access-9dvcw\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.679636 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.679462    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.679636 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.679532    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.679636 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.679622    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.679809 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.679641    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.679809 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.679777    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.681726 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.681691    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.681814 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.681782    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.701537 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.701511    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.701741 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.701721    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dvcw\" (UniqueName: \"kubernetes.io/projected/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-kube-api-access-9dvcw\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lll9m\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:50.813473 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:50.813414    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:51.133707 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:51.133678    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"]
Oct 30 04:25:51.135812 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:25:51.135785    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b928ba2_aa86_4b4b_89b4_1d6c8b6b1cc1.slice/crio-4ff3d9bd41dacfb0deafb8a780208cb0800fd9c1414c9c85f43406cb1490e1f5 WatchSource:0}: Error finding container 4ff3d9bd41dacfb0deafb8a780208cb0800fd9c1414c9c85f43406cb1490e1f5: Status 404 returned error can't find the container with id 4ff3d9bd41dacfb0deafb8a780208cb0800fd9c1414c9c85f43406cb1490e1f5
Oct 30 04:25:51.138156 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:51.138128    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:25:51.138219 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:51.138188    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:25:51.138283 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:51.138225    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:25:51.444765 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:51.444687    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m" event={"ID":"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1","Type":"ContainerStarted","Data":"8f6812d18fb2347359ebaca20b438759a88954b5e44a28b32681a95b2df8301c"}
Oct 30 04:25:51.444891 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:51.444768    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m" event={"ID":"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1","Type":"ContainerStarted","Data":"4ff3d9bd41dacfb0deafb8a780208cb0800fd9c1414c9c85f43406cb1490e1f5"}
Oct 30 04:25:51.488646 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:51.488582    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m" podStartSLOduration=1.488554822 podStartE2EDuration="1.488554822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:25:51.487209921 +0000 UTC m=+1670.899605493" watchObservedRunningTime="2026-10-30 04:25:51.488554822 +0000 UTC m=+1670.900950393"
Oct 30 04:25:51.813964 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:51.813923    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:52.815432 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:52.815388    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lll9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 04:25:52.815886 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:52.815447    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m" podUID="0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 04:25:52.889570 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:52.889534    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8"]
Oct 30 04:25:52.890489 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:52.890470    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b28e83-cdc5-4591-858b-80df98885d4e" containerName="istio-proxy"
Oct 30 04:25:52.894343 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:52.894323    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8"
Oct 30 04:25:52.906175 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:52.906152    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8"]
Oct 30 04:25:52.915447 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:52.915423    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1a0d3427\""
Oct 30 04:25:52.997489 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:52.997444    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1a0d3427-dockercfg-mcg4g\""
Oct 30 04:25:53.010089 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:53.010054    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6"]
Oct 30 04:25:53.036220 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:53.035846    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6"
Oct 30 04:25:53.101353 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:53.101205    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htsr8\" (UniqueName: \"kubernetes.io/projected/37adc92c-e067-479f-b982-c03987b298a2-kube-api-access-htsr8\") pod \"payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8\" (UID: \"37adc92c-e067-479f-b982-c03987b298a2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8"
Oct 30 04:25:53.117396 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:53.117292    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6"]
Oct 30 04:25:53.202554 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:53.202492    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37adc92c-e067-479f-b982-c03987b298a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8\" (UID: \"37adc92c-e067-479f-b982-c03987b298a2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8"
Oct 30 04:25:53.303657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:53.303620    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4-plugins-config-volume\") pod \"payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6\" (UID: \"bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4\") " pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6"
Oct 30 04:25:53.303832 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:53.303678    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjwqk\" (UniqueName: \"kubernetes.io/projected/bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4-kube-api-access-sjwqk\") pod \"payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6\" (UID: \"bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4\") " pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6"
Oct 30 04:25:53.303832 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:53.303801    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htsr8\" (UniqueName: \"kubernetes.io/projected/37adc92c-e067-479f-b982-c03987b298a2-kube-api-access-htsr8\") pod \"payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8\" (UID: \"37adc92c-e067-479f-b982-c03987b298a2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8"
Oct 30 04:25:53.303945 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:53.303883    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37adc92c-e067-479f-b982-c03987b298a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8\" (UID: \"37adc92c-e067-479f-b982-c03987b298a2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8"
Oct 30 04:25:53.304631 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:53.304611    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37adc92c-e067-479f-b982-c03987b298a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8\" (UID: \"37adc92c-e067-479f-b982-c03987b298a2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8"
Oct 30 04:25:53.327881 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:53.327823    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htsr8\" (UniqueName: \"kubernetes.io/projected/37adc92c-e067-479f-b982-c03987b298a2-kube-api-access-htsr8\") pod \"payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8\" (UID: \"37adc92c-e067-479f-b982-c03987b298a2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8"
Oct 30 04:25:53.405509 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:53.405476    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4-plugins-config-volume\") pod \"payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6\" (UID: \"bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4\") " pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6"
Oct 30 04:25:53.405634 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:53.405532    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjwqk\" (UniqueName: \"kubernetes.io/projected/bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4-kube-api-access-sjwqk\") pod \"payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6\" (UID: \"bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4\") " pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6"
Oct 30 04:25:53.406153 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:53.406133    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4-plugins-config-volume\") pod \"payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6\" (UID: \"bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4\") " pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6"
Oct 30 04:25:53.427901 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:53.427878    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjwqk\" (UniqueName: \"kubernetes.io/projected/bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4-kube-api-access-sjwqk\") pod \"payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6\" (UID: \"bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4\") " pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6"
Oct 30 04:25:53.532448 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:53.532410    4874 patch_prober.go:29] interesting pod/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 30 04:25:53.532630 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:53.532470    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh" podUID="32b4cf74-389e-4749-bf55-a1c52313e609" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:25:53.532630 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:53.532493    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8"
Oct 30 04:25:53.699368 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:53.699332    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6"
Oct 30 04:25:53.813957 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:53.813924    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lll9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 04:25:53.814114 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:53.813987    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m" podUID="0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 04:25:53.907299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:53.907059    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8"]
Oct 30 04:25:53.910232 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:25:53.910195    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37adc92c_e067_479f_b982_c03987b298a2.slice/crio-b7a7d84d3db3f4dcc546e7843487b2922f83052cd9af3ce0ceab0d876a58dbe4 WatchSource:0}: Error finding container b7a7d84d3db3f4dcc546e7843487b2922f83052cd9af3ce0ceab0d876a58dbe4: Status 404 returned error can't find the container with id b7a7d84d3db3f4dcc546e7843487b2922f83052cd9af3ce0ceab0d876a58dbe4
Oct 30 04:25:54.036709 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:54.036684    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6"]
Oct 30 04:25:54.038784 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:25:54.038758    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7f3ecc_4de5_4bb3_ab57_596f594c1bc4.slice/crio-303884e661fa2ace6f018ce907ee51dd7168df5d0bd352fa0ce58526b002415c WatchSource:0}: Error finding container 303884e661fa2ace6f018ce907ee51dd7168df5d0bd352fa0ce58526b002415c: Status 404 returned error can't find the container with id 303884e661fa2ace6f018ce907ee51dd7168df5d0bd352fa0ce58526b002415c
Oct 30 04:25:54.462545 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:54.462505    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6" event={"ID":"bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4","Type":"ContainerStarted","Data":"2bbf3826e873814a098981de7a1866f22268850a523be204bd1f9bbd858c598a"}
Oct 30 04:25:54.462696 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:54.462599    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6"
Oct 30 04:25:54.462696 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:54.462620    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6" event={"ID":"bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4","Type":"ContainerStarted","Data":"303884e661fa2ace6f018ce907ee51dd7168df5d0bd352fa0ce58526b002415c"}
Oct 30 04:25:54.464187 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:54.464151    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8" event={"ID":"37adc92c-e067-479f-b982-c03987b298a2","Type":"ContainerStarted","Data":"6b62fb2f5dd91ba3ab2636b015dbfd2bed625340d1895c65d7b0fd13e45790e1"}
Oct 30 04:25:54.464347 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:54.464211    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8" event={"ID":"37adc92c-e067-479f-b982-c03987b298a2","Type":"ContainerStarted","Data":"b7a7d84d3db3f4dcc546e7843487b2922f83052cd9af3ce0ceab0d876a58dbe4"}
Oct 30 04:25:54.464347 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:54.464292    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8"
Oct 30 04:25:54.513049 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:54.512982    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6" podStartSLOduration=2.512964644 podStartE2EDuration="2.512964644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:25:54.511518314 +0000 UTC m=+1673.923913888" watchObservedRunningTime="2026-10-30 04:25:54.512964644 +0000 UTC m=+1673.925360220"
Oct 30 04:25:54.554666 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:54.554605    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8" podStartSLOduration=2.5545863349999998 podStartE2EDuration="2.554586335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:25:54.552185582 +0000 UTC m=+1673.964581155" watchObservedRunningTime="2026-10-30 04:25:54.554586335 +0000 UTC m=+1673.966981911"
Oct 30 04:25:54.838264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:54.834291    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lll9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 04:25:54.858262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:54.843382    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m" podUID="0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 04:25:54.918287 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:54.918177    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:25:55.035271 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.033812    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-istio-data\") pod \"32b4cf74-389e-4749-bf55-a1c52313e609\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") "
Oct 30 04:25:55.035271 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.033902    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-istio-envoy\") pod \"32b4cf74-389e-4749-bf55-a1c52313e609\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") "
Oct 30 04:25:55.035271 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.033961    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32b4cf74-389e-4749-bf55-a1c52313e609-istio-token\" (UniqueName: \"kubernetes.io/projected/32b4cf74-389e-4749-bf55-a1c52313e609-istio-token\") pod \"32b4cf74-389e-4749-bf55-a1c52313e609\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") "
Oct 30 04:25:55.035271 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.033991    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-workload-socket\") pod \"32b4cf74-389e-4749-bf55-a1c52313e609\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") "
Oct 30 04:25:55.035271 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.034044    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32b4cf74-389e-4749-bf55-a1c52313e609-kube-api-access-pnq5w\" (UniqueName: \"kubernetes.io/projected/32b4cf74-389e-4749-bf55-a1c52313e609-kube-api-access-pnq5w\") pod \"32b4cf74-389e-4749-bf55-a1c52313e609\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") "
Oct 30 04:25:55.035271 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.034284    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-credential-socket\") pod \"32b4cf74-389e-4749-bf55-a1c52313e609\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") "
Oct 30 04:25:55.035271 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.034344    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-workload-certs\") pod \"32b4cf74-389e-4749-bf55-a1c52313e609\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") "
Oct 30 04:25:55.035271 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.034456    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32b4cf74-389e-4749-bf55-a1c52313e609-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32b4cf74-389e-4749-bf55-a1c52313e609-istiod-ca-cert\") pod \"32b4cf74-389e-4749-bf55-a1c52313e609\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") "
Oct 30 04:25:55.035271 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.034505    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32b4cf74-389e-4749-bf55-a1c52313e609-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32b4cf74-389e-4749-bf55-a1c52313e609-istio-podinfo\") pod \"32b4cf74-389e-4749-bf55-a1c52313e609\" (UID: \"32b4cf74-389e-4749-bf55-a1c52313e609\") "
Oct 30 04:25:55.035271 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.034994    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-credential-socket" pod "32b4cf74-389e-4749-bf55-a1c52313e609" (UID: "32b4cf74-389e-4749-bf55-a1c52313e609"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:25:55.035931 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.035232    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-workload-socket" pod "32b4cf74-389e-4749-bf55-a1c52313e609" (UID: "32b4cf74-389e-4749-bf55-a1c52313e609"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:25:55.035931 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.035336    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-workload-certs" pod "32b4cf74-389e-4749-bf55-a1c52313e609" (UID: "32b4cf74-389e-4749-bf55-a1c52313e609"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:25:55.035931 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.035679    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b4cf74-389e-4749-bf55-a1c52313e609-istiod-ca-cert" pod "32b4cf74-389e-4749-bf55-a1c52313e609" (UID: "32b4cf74-389e-4749-bf55-a1c52313e609"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:25:55.036311 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.036283    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-istio-data" pod "32b4cf74-389e-4749-bf55-a1c52313e609" (UID: "32b4cf74-389e-4749-bf55-a1c52313e609"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:25:55.038051 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.038028    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b4cf74-389e-4749-bf55-a1c52313e609-istio-token" pod "32b4cf74-389e-4749-bf55-a1c52313e609" (UID: "32b4cf74-389e-4749-bf55-a1c52313e609"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:25:55.038051 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.038049    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b4cf74-389e-4749-bf55-a1c52313e609-kube-api-access-pnq5w" pod "32b4cf74-389e-4749-bf55-a1c52313e609" (UID: "32b4cf74-389e-4749-bf55-a1c52313e609"). InnerVolumeSpecName "kube-api-access-pnq5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:25:55.038576 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.038551    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-istio-envoy" pod "32b4cf74-389e-4749-bf55-a1c52313e609" (UID: "32b4cf74-389e-4749-bf55-a1c52313e609"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:25:55.038936 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.038914    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32b4cf74-389e-4749-bf55-a1c52313e609-istio-podinfo" pod "32b4cf74-389e-4749-bf55-a1c52313e609" (UID: "32b4cf74-389e-4749-bf55-a1c52313e609"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:25:55.136153 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.136120    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:55.136153 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.136151    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b4cf74-389e-4749-bf55-a1c52313e609-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:55.136153 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.136161    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:55.136407 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.136173    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnq5w\" (UniqueName: \"kubernetes.io/projected/32b4cf74-389e-4749-bf55-a1c52313e609-kube-api-access-pnq5w\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:55.136407 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.136184    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:55.136407 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.136192    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:55.136407 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.136200    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32b4cf74-389e-4749-bf55-a1c52313e609-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:55.136407 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.136208    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32b4cf74-389e-4749-bf55-a1c52313e609-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:55.136407 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.136217    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32b4cf74-389e-4749-bf55-a1c52313e609-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:25:55.498102 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.498068    4874 generic.go:356] "Generic (PLEG): container finished" podID="32b4cf74-389e-4749-bf55-a1c52313e609" containerID="0bfbecdd7ea4eb2cb6aa00c75489eaf9987ae54867b4b61c7abcfbadec2d2f8b" exitCode=0
Oct 30 04:25:55.498306 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.498125    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"
Oct 30 04:25:55.498306 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.498144    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh" event={"ID":"32b4cf74-389e-4749-bf55-a1c52313e609","Type":"ContainerDied","Data":"0bfbecdd7ea4eb2cb6aa00c75489eaf9987ae54867b4b61c7abcfbadec2d2f8b"}
Oct 30 04:25:55.498306 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.498198    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh" event={"ID":"32b4cf74-389e-4749-bf55-a1c52313e609","Type":"ContainerDied","Data":"5506d485dc0a165259f654d959f50e20684d6bbf5453de08e8382a21cc7babc5"}
Oct 30 04:25:55.498306 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.498230    4874 scope.go:122] "RemoveContainer" containerID="0bfbecdd7ea4eb2cb6aa00c75489eaf9987ae54867b4b61c7abcfbadec2d2f8b"
Oct 30 04:25:55.508096 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.508079    4874 scope.go:122] "RemoveContainer" containerID="eb093c1b82de3a92d2c5265b0ef0d70c2ffe1e5073602d271324afc37aa33a5a"
Oct 30 04:25:55.516476 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.516455    4874 scope.go:122] "RemoveContainer" containerID="0bfbecdd7ea4eb2cb6aa00c75489eaf9987ae54867b4b61c7abcfbadec2d2f8b"
Oct 30 04:25:55.516715 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:25:55.516697    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bfbecdd7ea4eb2cb6aa00c75489eaf9987ae54867b4b61c7abcfbadec2d2f8b\": container with ID starting with 0bfbecdd7ea4eb2cb6aa00c75489eaf9987ae54867b4b61c7abcfbadec2d2f8b not found: ID does not exist" containerID="0bfbecdd7ea4eb2cb6aa00c75489eaf9987ae54867b4b61c7abcfbadec2d2f8b"
Oct 30 04:25:55.516792 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.516729    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bfbecdd7ea4eb2cb6aa00c75489eaf9987ae54867b4b61c7abcfbadec2d2f8b"} err="failed to get container status \"0bfbecdd7ea4eb2cb6aa00c75489eaf9987ae54867b4b61c7abcfbadec2d2f8b\": rpc error: code = NotFound desc = could not find container \"0bfbecdd7ea4eb2cb6aa00c75489eaf9987ae54867b4b61c7abcfbadec2d2f8b\": container with ID starting with 0bfbecdd7ea4eb2cb6aa00c75489eaf9987ae54867b4b61c7abcfbadec2d2f8b not found: ID does not exist"
Oct 30 04:25:55.516792 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.516751    4874 scope.go:122] "RemoveContainer" containerID="eb093c1b82de3a92d2c5265b0ef0d70c2ffe1e5073602d271324afc37aa33a5a"
Oct 30 04:25:55.517012 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:25:55.516993    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb093c1b82de3a92d2c5265b0ef0d70c2ffe1e5073602d271324afc37aa33a5a\": container with ID starting with eb093c1b82de3a92d2c5265b0ef0d70c2ffe1e5073602d271324afc37aa33a5a not found: ID does not exist" containerID="eb093c1b82de3a92d2c5265b0ef0d70c2ffe1e5073602d271324afc37aa33a5a"
Oct 30 04:25:55.517054 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.517018    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb093c1b82de3a92d2c5265b0ef0d70c2ffe1e5073602d271324afc37aa33a5a"} err="failed to get container status \"eb093c1b82de3a92d2c5265b0ef0d70c2ffe1e5073602d271324afc37aa33a5a\": rpc error: code = NotFound desc = could not find container \"eb093c1b82de3a92d2c5265b0ef0d70c2ffe1e5073602d271324afc37aa33a5a\": container with ID starting with eb093c1b82de3a92d2c5265b0ef0d70c2ffe1e5073602d271324afc37aa33a5a not found: ID does not exist"
Oct 30 04:25:55.548640 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.548611    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"]
Oct 30 04:25:55.559894 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.559870    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0c530e-openshift-default-65544d5499-vz2fh"]
Oct 30 04:25:55.617626 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:25:55.617592    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:25:55.813879 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.813797    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lll9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 04:25:55.813879 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:55.813856    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m" podUID="0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 04:25:56.729007 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:56.728975    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32b4cf74-389e-4749-bf55-a1c52313e609" path="/var/lib/kubelet/pods/32b4cf74-389e-4749-bf55-a1c52313e609/volumes"
Oct 30 04:25:56.813844 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:56.813813    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lll9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 04:25:56.814030 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:56.813907    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m" podUID="0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 04:25:57.814745 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:57.814715    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lll9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 04:25:57.815209 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:57.814811    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m" podUID="0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 04:25:58.196308 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:58.196216    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6"]
Oct 30 04:25:58.196868 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:58.196848    4874 cpu_manager.go:414] "containerMap: removing container" podUID="32b4cf74-389e-4749-bf55-a1c52313e609" containerName="istio-proxy"
Oct 30 04:25:58.196968 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:58.196869    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b4cf74-389e-4749-bf55-a1c52313e609" containerName="istio-proxy"
Oct 30 04:25:58.196968 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:58.196886    4874 cpu_manager.go:414] "containerMap: removing container" podUID="32b4cf74-389e-4749-bf55-a1c52313e609" containerName="istio-proxy"
Oct 30 04:25:58.196968 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:58.196894    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b4cf74-389e-4749-bf55-a1c52313e609" containerName="istio-proxy"
Oct 30 04:25:58.197115 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:58.197039    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b4cf74-389e-4749-bf55-a1c52313e609" containerName="istio-proxy"
Oct 30 04:25:58.197115 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:58.197053    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b4cf74-389e-4749-bf55-a1c52313e609" containerName="istio-proxy"
Oct 30 04:25:58.201888 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:58.201868    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6"
Oct 30 04:25:58.221265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:58.221226    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6"]
Oct 30 04:25:58.229490 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:58.229465    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1a0d3427-serving-cert\""
Oct 30 04:25:58.272854 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:58.272819    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da1b9a2c-09c4-4115-b48d-8d6ae387561c-metrics-certs\") pod \"maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6\" (UID: \"da1b9a2c-09c4-4115-b48d-8d6ae387561c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6"
Oct 30 04:25:58.273017 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:58.272875    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbhd7\" (UniqueName: \"kubernetes.io/projected/da1b9a2c-09c4-4115-b48d-8d6ae387561c-kube-api-access-wbhd7\") pod \"maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6\" (UID: \"da1b9a2c-09c4-4115-b48d-8d6ae387561c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6"
Oct 30 04:25:58.273069 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:58.273011    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da1b9a2c-09c4-4115-b48d-8d6ae387561c-openshift-service-ca\") pod \"maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6\" (UID: \"da1b9a2c-09c4-4115-b48d-8d6ae387561c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6"
Oct 30 04:25:58.273127 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:58.273081    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da1b9a2c-09c4-4115-b48d-8d6ae387561c-maas-api-tls\") pod \"maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6\" (UID: \"da1b9a2c-09c4-4115-b48d-8d6ae387561c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6"
Oct 30 04:25:58.374756 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:58.374721    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da1b9a2c-09c4-4115-b48d-8d6ae387561c-metrics-certs\") pod \"maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6\" (UID: \"da1b9a2c-09c4-4115-b48d-8d6ae387561c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6"
Oct 30 04:25:58.374950 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:58.374774    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbhd7\" (UniqueName: \"kubernetes.io/projected/da1b9a2c-09c4-4115-b48d-8d6ae387561c-kube-api-access-wbhd7\") pod \"maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6\" (UID: \"da1b9a2c-09c4-4115-b48d-8d6ae387561c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6"
Oct 30 04:25:58.374950 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:58.374893    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da1b9a2c-09c4-4115-b48d-8d6ae387561c-openshift-service-ca\") pod \"maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6\" (UID: \"da1b9a2c-09c4-4115-b48d-8d6ae387561c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6"
Oct 30 04:25:58.375084 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:58.374988    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da1b9a2c-09c4-4115-b48d-8d6ae387561c-maas-api-tls\") pod \"maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6\" (UID: \"da1b9a2c-09c4-4115-b48d-8d6ae387561c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6"
Oct 30 04:25:58.375170 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:25:58.375141    4874 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-serving-cert: secret "maas-api-e2e-mt-1a0d3427-serving-cert" not found
Oct 30 04:25:58.375230 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:25:58.375222    4874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da1b9a2c-09c4-4115-b48d-8d6ae387561c-maas-api-tls podName:da1b9a2c-09c4-4115-b48d-8d6ae387561c nodeName:}" failed. No retries permitted until 2026-10-30 04:25:58.875201175 +0000 UTC m=+1678.287596748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da1b9a2c-09c4-4115-b48d-8d6ae387561c-maas-api-tls") pod "maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6" (UID: "da1b9a2c-09c4-4115-b48d-8d6ae387561c") : secret "maas-api-e2e-mt-1a0d3427-serving-cert" not found
Oct 30 04:25:58.375671 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:58.375655    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da1b9a2c-09c4-4115-b48d-8d6ae387561c-openshift-service-ca\") pod \"maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6\" (UID: \"da1b9a2c-09c4-4115-b48d-8d6ae387561c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6"
Oct 30 04:25:58.377212 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:58.377189    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da1b9a2c-09c4-4115-b48d-8d6ae387561c-metrics-certs\") pod \"maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6\" (UID: \"da1b9a2c-09c4-4115-b48d-8d6ae387561c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6"
Oct 30 04:25:58.429451 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:58.429417    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbhd7\" (UniqueName: \"kubernetes.io/projected/da1b9a2c-09c4-4115-b48d-8d6ae387561c-kube-api-access-wbhd7\") pod \"maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6\" (UID: \"da1b9a2c-09c4-4115-b48d-8d6ae387561c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6"
Oct 30 04:25:58.881598 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:58.881557    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da1b9a2c-09c4-4115-b48d-8d6ae387561c-maas-api-tls\") pod \"maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6\" (UID: \"da1b9a2c-09c4-4115-b48d-8d6ae387561c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6"
Oct 30 04:25:58.884065 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:58.884038    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da1b9a2c-09c4-4115-b48d-8d6ae387561c-maas-api-tls\") pod \"maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6\" (UID: \"da1b9a2c-09c4-4115-b48d-8d6ae387561c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6"
Oct 30 04:25:59.125261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:59.122472    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6"
Oct 30 04:25:59.514233 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:59.514200    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6"]
Oct 30 04:25:59.517694 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:25:59.517671    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda1b9a2c_09c4_4115_b48d_8d6ae387561c.slice/crio-df8a26dff4908229204acee837aca47f889e249b40a2ea96099fdaf509fdb7dc WatchSource:0}: Error finding container df8a26dff4908229204acee837aca47f889e249b40a2ea96099fdaf509fdb7dc: Status 404 returned error can't find the container with id df8a26dff4908229204acee837aca47f889e249b40a2ea96099fdaf509fdb7dc
Oct 30 04:25:59.627849 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:59.627817    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:59.628318 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:59.628275    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:25:59.629312 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:25:59.629291    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:26:00.538449 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:00.538397    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6" event={"ID":"da1b9a2c-09c4-4115-b48d-8d6ae387561c","Type":"ContainerStarted","Data":"81d1b2bcb372683d9840492327082ca9dd6dd716295587d6c04d2af9283809c9"}
Oct 30 04:26:00.538943 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:00.538494    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6" event={"ID":"da1b9a2c-09c4-4115-b48d-8d6ae387561c","Type":"ContainerStarted","Data":"df8a26dff4908229204acee837aca47f889e249b40a2ea96099fdaf509fdb7dc"}
Oct 30 04:26:00.538943 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:00.538706    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6"
Oct 30 04:26:00.582722 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:00.582668    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6" podStartSLOduration=2.157403301 podStartE2EDuration="2.582654298s" totalImagesPullingTime="425.250997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:26:00.579973552 +0000 UTC m=+1679.992369348" watchObservedRunningTime="2026-10-30 04:26:00.582654298 +0000 UTC m=+1679.995049914"
Oct 30 04:26:00.912893 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:00.912814    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l"]
Oct 30 04:26:05.532701 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:05.531753    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8"
Oct 30 04:26:05.533866 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:05.533844    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6"
Oct 30 04:26:05.916395 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:05.916294    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l" podUID="2e928851-f639-4c63-bf99-3243a4f50edd" containerName="payload-processing" containerID="cri-o://f7430c704f554205c9b55663939282ba8dcf6ff311803cde00e1c4649729d50f" gracePeriod=25
Oct 30 04:26:06.203891 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:06.203864    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l"
Oct 30 04:26:06.270958 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:06.270837    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e928851-f639-4c63-bf99-3243a4f50edd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e928851-f639-4c63-bf99-3243a4f50edd-plugins-config-volume\") pod \"2e928851-f639-4c63-bf99-3243a4f50edd\" (UID: \"2e928851-f639-4c63-bf99-3243a4f50edd\") "
Oct 30 04:26:06.270958 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:06.270911    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e928851-f639-4c63-bf99-3243a4f50edd-kube-api-access-ltpqm\" (UniqueName: \"kubernetes.io/projected/2e928851-f639-4c63-bf99-3243a4f50edd-kube-api-access-ltpqm\") pod \"2e928851-f639-4c63-bf99-3243a4f50edd\" (UID: \"2e928851-f639-4c63-bf99-3243a4f50edd\") "
Oct 30 04:26:06.272312 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:06.271725    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e928851-f639-4c63-bf99-3243a4f50edd-plugins-config-volume" pod "2e928851-f639-4c63-bf99-3243a4f50edd" (UID: "2e928851-f639-4c63-bf99-3243a4f50edd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:26:06.277268 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:06.274062    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e928851-f639-4c63-bf99-3243a4f50edd-kube-api-access-ltpqm" pod "2e928851-f639-4c63-bf99-3243a4f50edd" (UID: "2e928851-f639-4c63-bf99-3243a4f50edd"). InnerVolumeSpecName "kube-api-access-ltpqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:26:06.371977 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:06.371949    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e928851-f639-4c63-bf99-3243a4f50edd-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:26:06.371977 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:06.371973    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltpqm\" (UniqueName: \"kubernetes.io/projected/2e928851-f639-4c63-bf99-3243a4f50edd-kube-api-access-ltpqm\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:26:06.585588 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:06.585566    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6"
Oct 30 04:26:06.664381 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:06.664342    4874 generic.go:356] "Generic (PLEG): container finished" podID="2e928851-f639-4c63-bf99-3243a4f50edd" containerID="f7430c704f554205c9b55663939282ba8dcf6ff311803cde00e1c4649729d50f" exitCode=0
Oct 30 04:26:06.664580 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:06.664410    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l"
Oct 30 04:26:06.664580 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:06.664407    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l" event={"ID":"2e928851-f639-4c63-bf99-3243a4f50edd","Type":"ContainerDied","Data":"f7430c704f554205c9b55663939282ba8dcf6ff311803cde00e1c4649729d50f"}
Oct 30 04:26:06.664580 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:06.664538    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l" event={"ID":"2e928851-f639-4c63-bf99-3243a4f50edd","Type":"ContainerDied","Data":"bba7e4dc6957457350676188dc5e34d1670bb16baaf14c8a632d09a219bbc417"}
Oct 30 04:26:06.664580 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:06.664568    4874 scope.go:122] "RemoveContainer" containerID="f7430c704f554205c9b55663939282ba8dcf6ff311803cde00e1c4649729d50f"
Oct 30 04:26:06.676768 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:06.676746    4874 scope.go:122] "RemoveContainer" containerID="f7430c704f554205c9b55663939282ba8dcf6ff311803cde00e1c4649729d50f"
Oct 30 04:26:06.677066 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:26:06.677047    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7430c704f554205c9b55663939282ba8dcf6ff311803cde00e1c4649729d50f\": container with ID starting with f7430c704f554205c9b55663939282ba8dcf6ff311803cde00e1c4649729d50f not found: ID does not exist" containerID="f7430c704f554205c9b55663939282ba8dcf6ff311803cde00e1c4649729d50f"
Oct 30 04:26:06.677140 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:06.677073    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7430c704f554205c9b55663939282ba8dcf6ff311803cde00e1c4649729d50f"} err="failed to get container status \"f7430c704f554205c9b55663939282ba8dcf6ff311803cde00e1c4649729d50f\": rpc error: code = NotFound desc = could not find container \"f7430c704f554205c9b55663939282ba8dcf6ff311803cde00e1c4649729d50f\": container with ID starting with f7430c704f554205c9b55663939282ba8dcf6ff311803cde00e1c4649729d50f not found: ID does not exist"
Oct 30 04:26:06.714852 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:06.714828    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l"]
Oct 30 04:26:06.728797 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:06.728777    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-397bda-65fb8d6656-ngr2l"]
Oct 30 04:26:07.131962 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:07.131932    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"]
Oct 30 04:26:07.132182 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:07.132136    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx" podUID="29a55637-b48a-4cc9-8b2a-454dacf07855" containerName="istio-proxy" containerID="cri-o://2c9f3c78bf1a5ee618f745a38d046ac17b9f637ea5b5f3e8a019c6f6d12ef2b8" gracePeriod=30
Oct 30 04:26:08.729310 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:08.729279    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e928851-f639-4c63-bf99-3243a4f50edd" path="/var/lib/kubelet/pods/2e928851-f639-4c63-bf99-3243a4f50edd/volumes"
Oct 30 04:26:12.395823 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.395766    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:26:12.601280 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.601006    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-token\" (UniqueName: \"kubernetes.io/projected/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-token\") pod \"29a55637-b48a-4cc9-8b2a-454dacf07855\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") "
Oct 30 04:26:12.601280 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.601058    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-data\") pod \"29a55637-b48a-4cc9-8b2a-454dacf07855\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") "
Oct 30 04:26:12.601280 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.601096    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-workload-certs\") pod \"29a55637-b48a-4cc9-8b2a-454dacf07855\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") "
Oct 30 04:26:12.601280 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.601170    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-credential-socket\") pod \"29a55637-b48a-4cc9-8b2a-454dacf07855\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") "
Oct 30 04:26:12.601280 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.601212    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-podinfo\") pod \"29a55637-b48a-4cc9-8b2a-454dacf07855\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") "
Oct 30 04:26:12.618857 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.618508    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-data" pod "29a55637-b48a-4cc9-8b2a-454dacf07855" (UID: "29a55637-b48a-4cc9-8b2a-454dacf07855"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:12.621840 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.621567    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-token" pod "29a55637-b48a-4cc9-8b2a-454dacf07855" (UID: "29a55637-b48a-4cc9-8b2a-454dacf07855"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:26:12.629028 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.624817    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-workload-certs" pod "29a55637-b48a-4cc9-8b2a-454dacf07855" (UID: "29a55637-b48a-4cc9-8b2a-454dacf07855"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:12.629028 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.626191    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-envoy\") pod \"29a55637-b48a-4cc9-8b2a-454dacf07855\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") "
Oct 30 04:26:12.629028 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.626251    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29a55637-b48a-4cc9-8b2a-454dacf07855-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29a55637-b48a-4cc9-8b2a-454dacf07855-istiod-ca-cert\") pod \"29a55637-b48a-4cc9-8b2a-454dacf07855\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") "
Oct 30 04:26:12.629028 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.626370    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-workload-socket\") pod \"29a55637-b48a-4cc9-8b2a-454dacf07855\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") "
Oct 30 04:26:12.629028 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.626422    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a55637-b48a-4cc9-8b2a-454dacf07855-kube-api-access-8x54r\" (UniqueName: \"kubernetes.io/projected/29a55637-b48a-4cc9-8b2a-454dacf07855-kube-api-access-8x54r\") pod \"29a55637-b48a-4cc9-8b2a-454dacf07855\" (UID: \"29a55637-b48a-4cc9-8b2a-454dacf07855\") "
Oct 30 04:26:12.629028 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.626913    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:26:12.629028 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.626929    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:26:12.629028 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.626942    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:26:12.630390 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.629964    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-workload-socket" pod "29a55637-b48a-4cc9-8b2a-454dacf07855" (UID: "29a55637-b48a-4cc9-8b2a-454dacf07855"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:12.630390 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.630357    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a55637-b48a-4cc9-8b2a-454dacf07855-istiod-ca-cert" pod "29a55637-b48a-4cc9-8b2a-454dacf07855" (UID: "29a55637-b48a-4cc9-8b2a-454dacf07855"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:26:12.645669 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.641477    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-credential-socket" pod "29a55637-b48a-4cc9-8b2a-454dacf07855" (UID: "29a55637-b48a-4cc9-8b2a-454dacf07855"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:12.650920 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.650839    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-envoy" pod "29a55637-b48a-4cc9-8b2a-454dacf07855" (UID: "29a55637-b48a-4cc9-8b2a-454dacf07855"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:12.651173 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.651141    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a55637-b48a-4cc9-8b2a-454dacf07855-kube-api-access-8x54r" pod "29a55637-b48a-4cc9-8b2a-454dacf07855" (UID: "29a55637-b48a-4cc9-8b2a-454dacf07855"). InnerVolumeSpecName "kube-api-access-8x54r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:26:12.655901 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.655866    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-podinfo" pod "29a55637-b48a-4cc9-8b2a-454dacf07855" (UID: "29a55637-b48a-4cc9-8b2a-454dacf07855"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:26:12.704908 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.704871    4874 generic.go:356] "Generic (PLEG): container finished" podID="29a55637-b48a-4cc9-8b2a-454dacf07855" containerID="2c9f3c78bf1a5ee618f745a38d046ac17b9f637ea5b5f3e8a019c6f6d12ef2b8" exitCode=0
Oct 30 04:26:12.705069 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.704918    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"
Oct 30 04:26:12.705069 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.704952    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx" event={"ID":"29a55637-b48a-4cc9-8b2a-454dacf07855","Type":"ContainerDied","Data":"2c9f3c78bf1a5ee618f745a38d046ac17b9f637ea5b5f3e8a019c6f6d12ef2b8"}
Oct 30 04:26:12.705069 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.705044    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx" event={"ID":"29a55637-b48a-4cc9-8b2a-454dacf07855","Type":"ContainerDied","Data":"7279b66d39af6491cfa0285bdf38f739caa1354c30ecf59cb940c26e3bf78db6"}
Oct 30 04:26:12.705173 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.705073    4874 scope.go:122] "RemoveContainer" containerID="2c9f3c78bf1a5ee618f745a38d046ac17b9f637ea5b5f3e8a019c6f6d12ef2b8"
Oct 30 04:26:12.715018 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.714992    4874 scope.go:122] "RemoveContainer" containerID="2c9f3c78bf1a5ee618f745a38d046ac17b9f637ea5b5f3e8a019c6f6d12ef2b8"
Oct 30 04:26:12.715349 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:26:12.715326    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c9f3c78bf1a5ee618f745a38d046ac17b9f637ea5b5f3e8a019c6f6d12ef2b8\": container with ID starting with 2c9f3c78bf1a5ee618f745a38d046ac17b9f637ea5b5f3e8a019c6f6d12ef2b8 not found: ID does not exist" containerID="2c9f3c78bf1a5ee618f745a38d046ac17b9f637ea5b5f3e8a019c6f6d12ef2b8"
Oct 30 04:26:12.715456 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.715359    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c9f3c78bf1a5ee618f745a38d046ac17b9f637ea5b5f3e8a019c6f6d12ef2b8"} err="failed to get container status \"2c9f3c78bf1a5ee618f745a38d046ac17b9f637ea5b5f3e8a019c6f6d12ef2b8\": rpc error: code = NotFound desc = could not find container \"2c9f3c78bf1a5ee618f745a38d046ac17b9f637ea5b5f3e8a019c6f6d12ef2b8\": container with ID starting with 2c9f3c78bf1a5ee618f745a38d046ac17b9f637ea5b5f3e8a019c6f6d12ef2b8 not found: ID does not exist"
Oct 30 04:26:12.727805 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.727781    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:26:12.727805 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.727801    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29a55637-b48a-4cc9-8b2a-454dacf07855-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:26:12.727984 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.727809    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:26:12.727984 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.727819    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x54r\" (UniqueName: \"kubernetes.io/projected/29a55637-b48a-4cc9-8b2a-454dacf07855-kube-api-access-8x54r\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:26:12.727984 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.727828    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a55637-b48a-4cc9-8b2a-454dacf07855-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:26:12.727984 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.727836    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29a55637-b48a-4cc9-8b2a-454dacf07855-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:26:12.789807 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.789777    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"]
Oct 30 04:26:12.817164 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:12.817138    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-8v6mx"]
Oct 30 04:26:14.023686 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.023332    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"]
Oct 30 04:26:14.024167 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.024119    4874 cpu_manager.go:414] "containerMap: removing container" podUID="29a55637-b48a-4cc9-8b2a-454dacf07855" containerName="istio-proxy"
Oct 30 04:26:14.024167 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.024138    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a55637-b48a-4cc9-8b2a-454dacf07855" containerName="istio-proxy"
Oct 30 04:26:14.024167 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.024155    4874 cpu_manager.go:414] "containerMap: removing container" podUID="2e928851-f639-4c63-bf99-3243a4f50edd" containerName="payload-processing"
Oct 30 04:26:14.024167 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.024164    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e928851-f639-4c63-bf99-3243a4f50edd" containerName="payload-processing"
Oct 30 04:26:14.024416 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.024294    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e928851-f639-4c63-bf99-3243a4f50edd" containerName="payload-processing"
Oct 30 04:26:14.024416 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.024319    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a55637-b48a-4cc9-8b2a-454dacf07855" containerName="istio-proxy"
Oct 30 04:26:14.034266 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.032357    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.069502 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.069471    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6dcea5-openshift-default-dockercfg-c4ngd\""
Oct 30 04:26:14.080370 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.080291    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"]
Oct 30 04:26:14.169815 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.169782    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istiod-ca-cert\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.169982 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.169822    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-token\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.169982 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.169913    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-workload-socket\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.170057 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.169985    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-workload-certs\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.170057 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.170020    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-data\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.170121 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.170073    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8llsq\" (UniqueName: \"kubernetes.io/projected/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-kube-api-access-8llsq\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.170153 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.170137    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-credential-socket\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.170187 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.170152    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-podinfo\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.170228 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.170185    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-envoy\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.271431 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.271391    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-token\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.271740 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.271705    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-workload-socket\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.271888 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.271800    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-workload-certs\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.271888 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.271855    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-data\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.272009 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.271891    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8llsq\" (UniqueName: \"kubernetes.io/projected/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-kube-api-access-8llsq\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.272009 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.271981    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-credential-socket\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.272009 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.272003    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-podinfo\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.272164 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.272053    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-envoy\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.272164 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.272101    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istiod-ca-cert\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.272395 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.272371    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-workload-certs\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.272561 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.272539    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-credential-socket\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.272883 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.272863    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-workload-socket\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.272970 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.272901    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-data\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.272970 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.272912    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istiod-ca-cert\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.276304 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.275374    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-envoy\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.276304 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.275880    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-podinfo\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.294939 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.294907    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-token\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.295135 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.295113    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8llsq\" (UniqueName: \"kubernetes.io/projected/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-kube-api-access-8llsq\") pod \"e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") " pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.374462 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.374433    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:14.693457 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.693375    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"]
Oct 30 04:26:14.696744 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:26:14.696716    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b33bf9_b5cb_473c_b6a4_d30dd097fd72.slice/crio-736692151e421b8190dc4f4ac4ea65addd2c86e2d2e159a992297d4838b18fa8 WatchSource:0}: Error finding container 736692151e421b8190dc4f4ac4ea65addd2c86e2d2e159a992297d4838b18fa8: Status 404 returned error can't find the container with id 736692151e421b8190dc4f4ac4ea65addd2c86e2d2e159a992297d4838b18fa8
Oct 30 04:26:14.699206 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.699176    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:26:14.699319 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.699262    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:26:14.699319 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.699295    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:26:14.719402 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.719378    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t" event={"ID":"30b33bf9-b5cb-473c-b6a4-d30dd097fd72","Type":"ContainerStarted","Data":"736692151e421b8190dc4f4ac4ea65addd2c86e2d2e159a992297d4838b18fa8"}
Oct 30 04:26:14.729915 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:14.729892    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29a55637-b48a-4cc9-8b2a-454dacf07855" path="/var/lib/kubelet/pods/29a55637-b48a-4cc9-8b2a-454dacf07855/volumes"
Oct 30 04:26:15.726847 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:15.726811    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t" event={"ID":"30b33bf9-b5cb-473c-b6a4-d30dd097fd72","Type":"ContainerStarted","Data":"67fe80662a55be354116d444a12147776a684fa5f7b0911fa6a40e67101995a2"}
Oct 30 04:26:15.793690 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:15.793645    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t" podStartSLOduration=2.793631732 podStartE2EDuration="2.793631732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:26:15.780363448 +0000 UTC m=+1695.192759022" watchObservedRunningTime="2026-10-30 04:26:15.793631732 +0000 UTC m=+1695.206027304"
Oct 30 04:26:16.375299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:16.375259    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:16.380698 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:16.380675    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:16.732133 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:16.732104    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:16.732946 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:16.732929    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:26:18.047869 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:18.047833    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:26:19.662976 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:19.662937    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11264-40b5de"]
Oct 30 04:26:19.669326 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:19.669298    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11264-40b5de"
Oct 30 04:26:19.700146 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:19.697659    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 04:26:19.700146 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:19.697903    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11264-40b5de"]
Oct 30 04:26:19.700146 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:19.697922    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 04:26:19.709316 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:19.709274    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gsgvm\""
Oct 30 04:26:19.784374 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:19.784341    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmsvk\" (UniqueName: \"kubernetes.io/projected/37aad952-570f-4255-9dd9-ce7f309d8273-kube-api-access-hmsvk\") pod \"test-curl-11264-40b5de\" (UID: \"37aad952-570f-4255-9dd9-ce7f309d8273\") " pod="opendatahub/test-curl-11264-40b5de"
Oct 30 04:26:19.885566 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:19.885527    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmsvk\" (UniqueName: \"kubernetes.io/projected/37aad952-570f-4255-9dd9-ce7f309d8273-kube-api-access-hmsvk\") pod \"test-curl-11264-40b5de\" (UID: \"37aad952-570f-4255-9dd9-ce7f309d8273\") " pod="opendatahub/test-curl-11264-40b5de"
Oct 30 04:26:19.907816 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:19.907789    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmsvk\" (UniqueName: \"kubernetes.io/projected/37aad952-570f-4255-9dd9-ce7f309d8273-kube-api-access-hmsvk\") pod \"test-curl-11264-40b5de\" (UID: \"37aad952-570f-4255-9dd9-ce7f309d8273\") " pod="opendatahub/test-curl-11264-40b5de"
Oct 30 04:26:20.013673 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:20.013647    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11264-40b5de"
Oct 30 04:26:20.325433 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:20.325407    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11264-40b5de"]
Oct 30 04:26:20.328394 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:26:20.328357    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37aad952_570f_4255_9dd9_ce7f309d8273.slice/crio-835e25f88cc83ad0d5a003a63c307126f698ad47ac88e4c126ccc01c5b3836d5 WatchSource:0}: Error finding container 835e25f88cc83ad0d5a003a63c307126f698ad47ac88e4c126ccc01c5b3836d5: Status 404 returned error can't find the container with id 835e25f88cc83ad0d5a003a63c307126f698ad47ac88e4c126ccc01c5b3836d5
Oct 30 04:26:20.827527 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:20.827491    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11264-40b5de" event={"ID":"37aad952-570f-4255-9dd9-ce7f309d8273","Type":"ContainerStarted","Data":"835e25f88cc83ad0d5a003a63c307126f698ad47ac88e4c126ccc01c5b3836d5"}
Oct 30 04:26:22.841049 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:22.841006    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11264-40b5de" event={"ID":"37aad952-570f-4255-9dd9-ce7f309d8273","Type":"ContainerStarted","Data":"4eeef6189767e78fa3a5187fb937f3c0e56af0d4df3dfb602137988ad6c80aeb"}
Oct 30 04:26:22.879804 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:22.879716    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11264-40b5de" podStartSLOduration=1.811177174 podStartE2EDuration="3.879696904s" totalImagesPullingTime="2.06851973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:26:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:26:22.876716021 +0000 UTC m=+1702.289111605" watchObservedRunningTime="2026-10-30 04:26:22.879696904 +0000 UTC m=+1702.292092478"
Oct 30 04:26:23.723768 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:23.723725    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11264-40b5de"]
Oct 30 04:26:23.727282 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:23.727199    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11264-40b5de"]
Oct 30 04:26:24.249216 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.249184    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"]
Oct 30 04:26:24.249656 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.249386    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5" podUID="c71f183b-8587-4f4f-a31f-203cb70db474" containerName="istio-proxy" containerID="cri-o://dfeaa2e75c1e0c5a7fa0079d5efe130ac969b8bd88f0a684b124967d7023a406" gracePeriod=30
Oct 30 04:26:24.491261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.490976    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65"]
Oct 30 04:26:24.495260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.491889    4874 cpu_manager.go:414] "containerMap: removing container" podUID="37aad952-570f-4255-9dd9-ce7f309d8273" containerName="test-curl-11264-40b5de"
Oct 30 04:26:24.495260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.491917    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37aad952-570f-4255-9dd9-ce7f309d8273" containerName="test-curl-11264-40b5de"
Oct 30 04:26:24.495260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.492052    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="37aad952-570f-4255-9dd9-ce7f309d8273" containerName="test-curl-11264-40b5de"
Oct 30 04:26:24.501652 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.501580    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65"
Oct 30 04:26:24.520178 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.520120    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-baa474\""
Oct 30 04:26:24.523501 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.523465    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65"]
Oct 30 04:26:24.565321 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.565282    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbpdd\" (UniqueName: \"kubernetes.io/projected/252d95cc-17a2-450d-b64f-29a1c0b48136-kube-api-access-qbpdd\") pod \"payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65\" (UID: \"252d95cc-17a2-450d-b64f-29a1c0b48136\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65"
Oct 30 04:26:24.565481 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.565458    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/252d95cc-17a2-450d-b64f-29a1c0b48136-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65\" (UID: \"252d95cc-17a2-450d-b64f-29a1c0b48136\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65"
Oct 30 04:26:24.578848 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.578818    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7"]
Oct 30 04:26:24.582536 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.582515    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7"
Oct 30 04:26:24.605526 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.605497    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7"]
Oct 30 04:26:24.667006 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.666968    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/252d95cc-17a2-450d-b64f-29a1c0b48136-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65\" (UID: \"252d95cc-17a2-450d-b64f-29a1c0b48136\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65"
Oct 30 04:26:24.667168 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.667018    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxf86\" (UniqueName: \"kubernetes.io/projected/ed9a9630-d7e9-4480-8139-0fcd83c56c32-kube-api-access-rxf86\") pod \"maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7\" (UID: \"ed9a9630-d7e9-4480-8139-0fcd83c56c32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7"
Oct 30 04:26:24.667224 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.667163    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbpdd\" (UniqueName: \"kubernetes.io/projected/252d95cc-17a2-450d-b64f-29a1c0b48136-kube-api-access-qbpdd\") pod \"payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65\" (UID: \"252d95cc-17a2-450d-b64f-29a1c0b48136\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65"
Oct 30 04:26:24.667224 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.667206    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9a9630-d7e9-4480-8139-0fcd83c56c32-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7\" (UID: \"ed9a9630-d7e9-4480-8139-0fcd83c56c32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7"
Oct 30 04:26:24.667355 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.667336    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed9a9630-d7e9-4480-8139-0fcd83c56c32-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7\" (UID: \"ed9a9630-d7e9-4480-8139-0fcd83c56c32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7"
Oct 30 04:26:24.667473 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.667433    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed9a9630-d7e9-4480-8139-0fcd83c56c32-metrics-certs\") pod \"maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7\" (UID: \"ed9a9630-d7e9-4480-8139-0fcd83c56c32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7"
Oct 30 04:26:24.667835 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.667814    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/252d95cc-17a2-450d-b64f-29a1c0b48136-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65\" (UID: \"252d95cc-17a2-450d-b64f-29a1c0b48136\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65"
Oct 30 04:26:24.681082 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.681060    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-bf4238-serving-cert\""
Oct 30 04:26:24.689923 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.689901    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbpdd\" (UniqueName: \"kubernetes.io/projected/252d95cc-17a2-450d-b64f-29a1c0b48136-kube-api-access-qbpdd\") pod \"payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65\" (UID: \"252d95cc-17a2-450d-b64f-29a1c0b48136\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65"
Oct 30 04:26:24.768449 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.768392    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxf86\" (UniqueName: \"kubernetes.io/projected/ed9a9630-d7e9-4480-8139-0fcd83c56c32-kube-api-access-rxf86\") pod \"maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7\" (UID: \"ed9a9630-d7e9-4480-8139-0fcd83c56c32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7"
Oct 30 04:26:24.768564 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.768506    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9a9630-d7e9-4480-8139-0fcd83c56c32-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7\" (UID: \"ed9a9630-d7e9-4480-8139-0fcd83c56c32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7"
Oct 30 04:26:24.768615 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.768584    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed9a9630-d7e9-4480-8139-0fcd83c56c32-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7\" (UID: \"ed9a9630-d7e9-4480-8139-0fcd83c56c32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7"
Oct 30 04:26:24.768664 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.768632    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed9a9630-d7e9-4480-8139-0fcd83c56c32-metrics-certs\") pod \"maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7\" (UID: \"ed9a9630-d7e9-4480-8139-0fcd83c56c32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7"
Oct 30 04:26:24.769131 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.769108    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9a9630-d7e9-4480-8139-0fcd83c56c32-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7\" (UID: \"ed9a9630-d7e9-4480-8139-0fcd83c56c32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7"
Oct 30 04:26:24.771043 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.771017    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed9a9630-d7e9-4480-8139-0fcd83c56c32-metrics-certs\") pod \"maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7\" (UID: \"ed9a9630-d7e9-4480-8139-0fcd83c56c32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7"
Oct 30 04:26:24.771043 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.771036    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed9a9630-d7e9-4480-8139-0fcd83c56c32-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7\" (UID: \"ed9a9630-d7e9-4480-8139-0fcd83c56c32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7"
Oct 30 04:26:24.794735 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.794709    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxf86\" (UniqueName: \"kubernetes.io/projected/ed9a9630-d7e9-4480-8139-0fcd83c56c32-kube-api-access-rxf86\") pod \"maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7\" (UID: \"ed9a9630-d7e9-4480-8139-0fcd83c56c32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7"
Oct 30 04:26:24.831882 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.831851    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65"
Oct 30 04:26:24.893977 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.893926    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11264-40b5de" podUID="37aad952-570f-4255-9dd9-ce7f309d8273" containerName="test-curl-11264-40b5de" containerID="cri-o://4eeef6189767e78fa3a5187fb937f3c0e56af0d4df3dfb602137988ad6c80aeb" gracePeriod=2
Oct 30 04:26:24.912829 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:24.912531    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7"
Oct 30 04:26:25.215555 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:25.215521    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65"]
Oct 30 04:26:25.216413 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:26:25.216384    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252d95cc_17a2_450d_b64f_29a1c0b48136.slice/crio-b18d08bc68b9afaced42c054eb378155511fd738cb45540805c6f05d96d1ea9a WatchSource:0}: Error finding container b18d08bc68b9afaced42c054eb378155511fd738cb45540805c6f05d96d1ea9a: Status 404 returned error can't find the container with id b18d08bc68b9afaced42c054eb378155511fd738cb45540805c6f05d96d1ea9a
Oct 30 04:26:25.266076 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:25.266051    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7"]
Oct 30 04:26:25.270277 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:26:25.270224    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded9a9630_d7e9_4480_8139_0fcd83c56c32.slice/crio-d3a860a831276e8fc563217d6856a19cf2c9311e5d450178265c28f8cac5427c WatchSource:0}: Error finding container d3a860a831276e8fc563217d6856a19cf2c9311e5d450178265c28f8cac5427c: Status 404 returned error can't find the container with id d3a860a831276e8fc563217d6856a19cf2c9311e5d450178265c28f8cac5427c
Oct 30 04:26:25.646588 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:26:25.646556    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:26:25.952033 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:25.951944    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65" event={"ID":"252d95cc-17a2-450d-b64f-29a1c0b48136","Type":"ContainerStarted","Data":"6ec9c06685e38a64b2a217d716770e9badfeded8dfb5739f5175428a380bbf0b"}
Oct 30 04:26:25.952282 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:25.952260    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65" event={"ID":"252d95cc-17a2-450d-b64f-29a1c0b48136","Type":"ContainerStarted","Data":"b18d08bc68b9afaced42c054eb378155511fd738cb45540805c6f05d96d1ea9a"}
Oct 30 04:26:25.954232 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:25.954198    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65"
Oct 30 04:26:25.955825 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:25.955803    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7" event={"ID":"ed9a9630-d7e9-4480-8139-0fcd83c56c32","Type":"ContainerStarted","Data":"e2d1257a7cc1f67d6b7a3a3040b523543b5a106fb3fc5e230aa88dcc80e57a3a"}
Oct 30 04:26:25.956949 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:25.956928    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7" event={"ID":"ed9a9630-d7e9-4480-8139-0fcd83c56c32","Type":"ContainerStarted","Data":"d3a860a831276e8fc563217d6856a19cf2c9311e5d450178265c28f8cac5427c"}
Oct 30 04:26:25.957118 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:25.957104    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7"
Oct 30 04:26:25.995213 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:25.995158    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65" podStartSLOduration=1.995145259 podStartE2EDuration="1.995145259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:26:25.993498463 +0000 UTC m=+1705.405894039" watchObservedRunningTime="2026-10-30 04:26:25.995145259 +0000 UTC m=+1705.407540832"
Oct 30 04:26:26.034542 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:26.034492    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7" podStartSLOduration=1.740036648 podStartE2EDuration="2.034479525s" totalImagesPullingTime="294.442877ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:26:26.03381657 +0000 UTC m=+1705.446212143" watchObservedRunningTime="2026-10-30 04:26:26.034479525 +0000 UTC m=+1705.446875098"
Oct 30 04:26:27.145662 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:27.145638    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11264-40b5de"
Oct 30 04:26:27.167048 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:27.167023    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37aad952-570f-4255-9dd9-ce7f309d8273-kube-api-access-hmsvk\" (UniqueName: \"kubernetes.io/projected/37aad952-570f-4255-9dd9-ce7f309d8273-kube-api-access-hmsvk\") pod \"37aad952-570f-4255-9dd9-ce7f309d8273\" (UID: \"37aad952-570f-4255-9dd9-ce7f309d8273\") "
Oct 30 04:26:27.168998 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:27.168979    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37aad952-570f-4255-9dd9-ce7f309d8273-kube-api-access-hmsvk" pod "37aad952-570f-4255-9dd9-ce7f309d8273" (UID: "37aad952-570f-4255-9dd9-ce7f309d8273"). InnerVolumeSpecName "kube-api-access-hmsvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:26:27.268545 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:27.268510    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmsvk\" (UniqueName: \"kubernetes.io/projected/37aad952-570f-4255-9dd9-ce7f309d8273-kube-api-access-hmsvk\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:26:27.968064 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:27.968034    4874 generic.go:356] "Generic (PLEG): container finished" podID="37aad952-570f-4255-9dd9-ce7f309d8273" containerID="4eeef6189767e78fa3a5187fb937f3c0e56af0d4df3dfb602137988ad6c80aeb" exitCode=137
Oct 30 04:26:27.968228 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:27.968073    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11264-40b5de"
Oct 30 04:26:27.968228 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:27.968127    4874 scope.go:122] "RemoveContainer" containerID="4eeef6189767e78fa3a5187fb937f3c0e56af0d4df3dfb602137988ad6c80aeb"
Oct 30 04:26:27.976765 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:27.976748    4874 scope.go:122] "RemoveContainer" containerID="4eeef6189767e78fa3a5187fb937f3c0e56af0d4df3dfb602137988ad6c80aeb"
Oct 30 04:26:27.977022 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:26:27.977004    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eeef6189767e78fa3a5187fb937f3c0e56af0d4df3dfb602137988ad6c80aeb\": container with ID starting with 4eeef6189767e78fa3a5187fb937f3c0e56af0d4df3dfb602137988ad6c80aeb not found: ID does not exist" containerID="4eeef6189767e78fa3a5187fb937f3c0e56af0d4df3dfb602137988ad6c80aeb"
Oct 30 04:26:27.977068 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:27.977031    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eeef6189767e78fa3a5187fb937f3c0e56af0d4df3dfb602137988ad6c80aeb"} err="failed to get container status \"4eeef6189767e78fa3a5187fb937f3c0e56af0d4df3dfb602137988ad6c80aeb\": rpc error: code = NotFound desc = could not find container \"4eeef6189767e78fa3a5187fb937f3c0e56af0d4df3dfb602137988ad6c80aeb\": container with ID starting with 4eeef6189767e78fa3a5187fb937f3c0e56af0d4df3dfb602137988ad6c80aeb not found: ID does not exist"
Oct 30 04:26:27.989222 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:26:27.989200    4874 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11264-40b5de\" is forbidden: User \"system:node:ip-10-0-138-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-160.ec2.internal' and this object" podUID="37aad952-570f-4255-9dd9-ce7f309d8273" pod="opendatahub/test-curl-11264-40b5de"
Oct 30 04:26:28.729511 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:28.729481    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37aad952-570f-4255-9dd9-ce7f309d8273" path="/var/lib/kubelet/pods/37aad952-570f-4255-9dd9-ce7f309d8273/volumes"
Oct 30 04:26:29.506820 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.506797    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:26:29.602954 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.602875    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c71f183b-8587-4f4f-a31f-203cb70db474-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c71f183b-8587-4f4f-a31f-203cb70db474-istio-podinfo\") pod \"c71f183b-8587-4f4f-a31f-203cb70db474\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") "
Oct 30 04:26:29.602954 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.602917    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-istio-envoy\") pod \"c71f183b-8587-4f4f-a31f-203cb70db474\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") "
Oct 30 04:26:29.602954 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.602942    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c71f183b-8587-4f4f-a31f-203cb70db474-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c71f183b-8587-4f4f-a31f-203cb70db474-istiod-ca-cert\") pod \"c71f183b-8587-4f4f-a31f-203cb70db474\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") "
Oct 30 04:26:29.603222 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.603063    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c71f183b-8587-4f4f-a31f-203cb70db474-istio-token\" (UniqueName: \"kubernetes.io/projected/c71f183b-8587-4f4f-a31f-203cb70db474-istio-token\") pod \"c71f183b-8587-4f4f-a31f-203cb70db474\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") "
Oct 30 04:26:29.603222 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.603089    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c71f183b-8587-4f4f-a31f-203cb70db474-kube-api-access-kfstt\" (UniqueName: \"kubernetes.io/projected/c71f183b-8587-4f4f-a31f-203cb70db474-kube-api-access-kfstt\") pod \"c71f183b-8587-4f4f-a31f-203cb70db474\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") "
Oct 30 04:26:29.603222 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.603152    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-credential-socket\") pod \"c71f183b-8587-4f4f-a31f-203cb70db474\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") "
Oct 30 04:26:29.603399 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.603254    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-workload-socket\") pod \"c71f183b-8587-4f4f-a31f-203cb70db474\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") "
Oct 30 04:26:29.603399 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.603293    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-istio-data\") pod \"c71f183b-8587-4f4f-a31f-203cb70db474\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") "
Oct 30 04:26:29.603399 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.603341    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-workload-certs\") pod \"c71f183b-8587-4f4f-a31f-203cb70db474\" (UID: \"c71f183b-8587-4f4f-a31f-203cb70db474\") "
Oct 30 04:26:29.603531 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.603479    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-credential-socket" pod "c71f183b-8587-4f4f-a31f-203cb70db474" (UID: "c71f183b-8587-4f4f-a31f-203cb70db474"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:29.603531 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.603513    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c71f183b-8587-4f4f-a31f-203cb70db474-istiod-ca-cert" pod "c71f183b-8587-4f4f-a31f-203cb70db474" (UID: "c71f183b-8587-4f4f-a31f-203cb70db474"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:26:29.603772 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.603675    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-workload-socket" pod "c71f183b-8587-4f4f-a31f-203cb70db474" (UID: "c71f183b-8587-4f4f-a31f-203cb70db474"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:29.603772 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.603730    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-istio-data" pod "c71f183b-8587-4f4f-a31f-203cb70db474" (UID: "c71f183b-8587-4f4f-a31f-203cb70db474"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:29.604016 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.603880    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-workload-certs" pod "c71f183b-8587-4f4f-a31f-203cb70db474" (UID: "c71f183b-8587-4f4f-a31f-203cb70db474"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:29.604108 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.604022    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c71f183b-8587-4f4f-a31f-203cb70db474-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.604108 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.604043    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.604108 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.604058    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.604108 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.604073    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.604108 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.604087    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.605666 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.605637    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71f183b-8587-4f4f-a31f-203cb70db474-kube-api-access-kfstt" pod "c71f183b-8587-4f4f-a31f-203cb70db474" (UID: "c71f183b-8587-4f4f-a31f-203cb70db474"). InnerVolumeSpecName "kube-api-access-kfstt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:26:29.605800 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.605765    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c71f183b-8587-4f4f-a31f-203cb70db474-istio-podinfo" pod "c71f183b-8587-4f4f-a31f-203cb70db474" (UID: "c71f183b-8587-4f4f-a31f-203cb70db474"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:26:29.605860 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.605833    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71f183b-8587-4f4f-a31f-203cb70db474-istio-token" pod "c71f183b-8587-4f4f-a31f-203cb70db474" (UID: "c71f183b-8587-4f4f-a31f-203cb70db474"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:26:29.606315 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.606298    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-istio-envoy" pod "c71f183b-8587-4f4f-a31f-203cb70db474" (UID: "c71f183b-8587-4f4f-a31f-203cb70db474"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:29.705395 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.705363    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c71f183b-8587-4f4f-a31f-203cb70db474-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.705395 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.705394    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c71f183b-8587-4f4f-a31f-203cb70db474-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.705589 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.705407    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c71f183b-8587-4f4f-a31f-203cb70db474-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.705589 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.705419    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfstt\" (UniqueName: \"kubernetes.io/projected/c71f183b-8587-4f4f-a31f-203cb70db474-kube-api-access-kfstt\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.980915 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.980883    4874 generic.go:356] "Generic (PLEG): container finished" podID="c71f183b-8587-4f4f-a31f-203cb70db474" containerID="dfeaa2e75c1e0c5a7fa0079d5efe130ac969b8bd88f0a684b124967d7023a406" exitCode=0
Oct 30 04:26:29.981348 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.980930    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"
Oct 30 04:26:29.981348 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.980948    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5" event={"ID":"c71f183b-8587-4f4f-a31f-203cb70db474","Type":"ContainerDied","Data":"dfeaa2e75c1e0c5a7fa0079d5efe130ac969b8bd88f0a684b124967d7023a406"}
Oct 30 04:26:29.981348 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.980985    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5" event={"ID":"c71f183b-8587-4f4f-a31f-203cb70db474","Type":"ContainerDied","Data":"96f3bf0f5611bcfc1c5ed794db2d52a08f63167ab7b0fa843e982fbe6e09144a"}
Oct 30 04:26:29.981348 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.981006    4874 scope.go:122] "RemoveContainer" containerID="dfeaa2e75c1e0c5a7fa0079d5efe130ac969b8bd88f0a684b124967d7023a406"
Oct 30 04:26:29.991049 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.991032    4874 scope.go:122] "RemoveContainer" containerID="dfeaa2e75c1e0c5a7fa0079d5efe130ac969b8bd88f0a684b124967d7023a406"
Oct 30 04:26:29.991353 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:26:29.991333    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfeaa2e75c1e0c5a7fa0079d5efe130ac969b8bd88f0a684b124967d7023a406\": container with ID starting with dfeaa2e75c1e0c5a7fa0079d5efe130ac969b8bd88f0a684b124967d7023a406 not found: ID does not exist" containerID="dfeaa2e75c1e0c5a7fa0079d5efe130ac969b8bd88f0a684b124967d7023a406"
Oct 30 04:26:29.991404 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:29.991361    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfeaa2e75c1e0c5a7fa0079d5efe130ac969b8bd88f0a684b124967d7023a406"} err="failed to get container status \"dfeaa2e75c1e0c5a7fa0079d5efe130ac969b8bd88f0a684b124967d7023a406\": rpc error: code = NotFound desc = could not find container \"dfeaa2e75c1e0c5a7fa0079d5efe130ac969b8bd88f0a684b124967d7023a406\": container with ID starting with dfeaa2e75c1e0c5a7fa0079d5efe130ac969b8bd88f0a684b124967d7023a406 not found: ID does not exist"
Oct 30 04:26:30.030940 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:30.030919    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"]
Oct 30 04:26:30.041908 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:30.041887    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-pvqj5"]
Oct 30 04:26:30.730735 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:30.730697    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c71f183b-8587-4f4f-a31f-203cb70db474" path="/var/lib/kubelet/pods/c71f183b-8587-4f4f-a31f-203cb70db474/volumes"
Oct 30 04:26:32.974348 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:32.974322    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7"
Oct 30 04:26:33.268306 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:33.263498    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:26:33.287551 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:33.287513    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg"]
Oct 30 04:26:33.288582 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:33.288556    4874 cpu_manager.go:414] "containerMap: removing container" podUID="c71f183b-8587-4f4f-a31f-203cb70db474" containerName="istio-proxy"
Oct 30 04:26:33.288744 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:33.288732    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c71f183b-8587-4f4f-a31f-203cb70db474" containerName="istio-proxy"
Oct 30 04:26:33.289002 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:33.288982    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="c71f183b-8587-4f4f-a31f-203cb70db474" containerName="istio-proxy"
Oct 30 04:26:33.331638 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:33.331611    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg"
Oct 30 04:26:33.333867 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:33.333843    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg"]
Oct 30 04:26:33.348523 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:33.348496    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-6dcea5-serving-cert\""
Oct 30 04:26:33.468561 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:33.468510    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swxcg\" (UniqueName: \"kubernetes.io/projected/9c67bdeb-196a-4314-a541-48cbac967a46-kube-api-access-swxcg\") pod \"maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg\" (UID: \"9c67bdeb-196a-4314-a541-48cbac967a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg"
Oct 30 04:26:33.468728 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:33.468579    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c67bdeb-196a-4314-a541-48cbac967a46-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg\" (UID: \"9c67bdeb-196a-4314-a541-48cbac967a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg"
Oct 30 04:26:33.468728 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:33.468666    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c67bdeb-196a-4314-a541-48cbac967a46-metrics-certs\") pod \"maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg\" (UID: \"9c67bdeb-196a-4314-a541-48cbac967a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg"
Oct 30 04:26:33.468900 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:33.468876    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c67bdeb-196a-4314-a541-48cbac967a46-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg\" (UID: \"9c67bdeb-196a-4314-a541-48cbac967a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg"
Oct 30 04:26:33.569867 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:33.569784    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c67bdeb-196a-4314-a541-48cbac967a46-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg\" (UID: \"9c67bdeb-196a-4314-a541-48cbac967a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg"
Oct 30 04:26:33.570043 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:33.569867    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swxcg\" (UniqueName: \"kubernetes.io/projected/9c67bdeb-196a-4314-a541-48cbac967a46-kube-api-access-swxcg\") pod \"maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg\" (UID: \"9c67bdeb-196a-4314-a541-48cbac967a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg"
Oct 30 04:26:33.570043 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:33.569900    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c67bdeb-196a-4314-a541-48cbac967a46-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg\" (UID: \"9c67bdeb-196a-4314-a541-48cbac967a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg"
Oct 30 04:26:33.570043 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:26:33.569926    4874 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-serving-cert: secret "maas-api-e2e-ipp-a-6dcea5-serving-cert" not found
Oct 30 04:26:33.570043 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:33.569940    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c67bdeb-196a-4314-a541-48cbac967a46-metrics-certs\") pod \"maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg\" (UID: \"9c67bdeb-196a-4314-a541-48cbac967a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg"
Oct 30 04:26:33.570043 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:26:33.570009    4874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c67bdeb-196a-4314-a541-48cbac967a46-maas-api-tls podName:9c67bdeb-196a-4314-a541-48cbac967a46 nodeName:}" failed. No retries permitted until 2026-10-30 04:26:34.069988568 +0000 UTC m=+1713.482384122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9c67bdeb-196a-4314-a541-48cbac967a46-maas-api-tls") pod "maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg" (UID: "9c67bdeb-196a-4314-a541-48cbac967a46") : secret "maas-api-e2e-ipp-a-6dcea5-serving-cert" not found
Oct 30 04:26:33.570586 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:33.570564    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c67bdeb-196a-4314-a541-48cbac967a46-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg\" (UID: \"9c67bdeb-196a-4314-a541-48cbac967a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg"
Oct 30 04:26:33.572434 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:33.572412    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c67bdeb-196a-4314-a541-48cbac967a46-metrics-certs\") pod \"maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg\" (UID: \"9c67bdeb-196a-4314-a541-48cbac967a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg"
Oct 30 04:26:33.592526 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:33.592500    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swxcg\" (UniqueName: \"kubernetes.io/projected/9c67bdeb-196a-4314-a541-48cbac967a46-kube-api-access-swxcg\") pod \"maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg\" (UID: \"9c67bdeb-196a-4314-a541-48cbac967a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg"
Oct 30 04:26:34.078056 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:34.078025    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c67bdeb-196a-4314-a541-48cbac967a46-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg\" (UID: \"9c67bdeb-196a-4314-a541-48cbac967a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg"
Oct 30 04:26:34.080213 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:34.080194    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c67bdeb-196a-4314-a541-48cbac967a46-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg\" (UID: \"9c67bdeb-196a-4314-a541-48cbac967a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg"
Oct 30 04:26:34.267009 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:34.265401    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg"
Oct 30 04:26:34.626892 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:34.626818    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg"]
Oct 30 04:26:34.629219 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:26:34.629194    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c67bdeb_196a_4314_a541_48cbac967a46.slice/crio-79af2f81be04dac4b3ea485cdefeeda36f4ad74dd14e239d19cf246c7589cf9e WatchSource:0}: Error finding container 79af2f81be04dac4b3ea485cdefeeda36f4ad74dd14e239d19cf246c7589cf9e: Status 404 returned error can't find the container with id 79af2f81be04dac4b3ea485cdefeeda36f4ad74dd14e239d19cf246c7589cf9e
Oct 30 04:26:35.010196 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:35.010156    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg" event={"ID":"9c67bdeb-196a-4314-a541-48cbac967a46","Type":"ContainerStarted","Data":"97bafc537f8289c7246aa3dd8b35d90bc2cb5e76e7e8c9ff6a2d76e746a7a021"}
Oct 30 04:26:35.010350 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:35.010220    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg" event={"ID":"9c67bdeb-196a-4314-a541-48cbac967a46","Type":"ContainerStarted","Data":"79af2f81be04dac4b3ea485cdefeeda36f4ad74dd14e239d19cf246c7589cf9e"}
Oct 30 04:26:35.010350 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:35.010271    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg"
Oct 30 04:26:35.052668 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:35.052619    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg" podStartSLOduration=1.7475753680000001 podStartE2EDuration="2.052604888s" totalImagesPullingTime="305.02952ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:26:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:26:35.05002992 +0000 UTC m=+1714.462425505" watchObservedRunningTime="2026-10-30 04:26:35.052604888 +0000 UTC m=+1714.465000461"
Oct 30 04:26:37.971595 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:37.971568    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65"
Oct 30 04:26:41.021295 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:41.021266    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg"
Oct 30 04:26:51.640076 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:26:51.640041    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:26:55.615540 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:26:55.615506    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:27:01.877703 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:01.877214    4874 patch_prober.go:29] interesting pod/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": read tcp 10.134.0.2:52180->10.134.0.76:15021: read: connection reset by peer" start-of-body=
Oct 30 04:27:01.877703 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:01.877284    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t" podUID="30b33bf9-b5cb-473c-b6a4-d30dd097fd72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": read tcp 10.134.0.2:52180->10.134.0.76:15021: read: connection reset by peer"
Oct 30 04:27:02.279784 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:02.279747    4874 generic.go:356] "Generic (PLEG): container finished" podID="30b33bf9-b5cb-473c-b6a4-d30dd097fd72" containerID="67fe80662a55be354116d444a12147776a684fa5f7b0911fa6a40e67101995a2" exitCode=0
Oct 30 04:27:02.280265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:02.280044    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t" event={"ID":"30b33bf9-b5cb-473c-b6a4-d30dd097fd72","Type":"ContainerDied","Data":"67fe80662a55be354116d444a12147776a684fa5f7b0911fa6a40e67101995a2"}
Oct 30 04:27:02.281382 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:02.281365    4874 scope.go:122] "RemoveContainer" containerID="67fe80662a55be354116d444a12147776a684fa5f7b0911fa6a40e67101995a2"
Oct 30 04:27:02.284930 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:02.284117    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:27:02.284930 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:02.284208    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:27:02.284930 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:02.284271    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:27:03.222821 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:03.222788    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65"]
Oct 30 04:27:03.315489 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:03.314964    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t" event={"ID":"30b33bf9-b5cb-473c-b6a4-d30dd097fd72","Type":"ContainerStarted","Data":"79befbb93d75436cb8048d2a297155598543a371632244b81b30319d1d8cecba"}
Oct 30 04:27:03.390006 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:03.389957    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:27:03.390229 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:03.390154    4874 patch_prober.go:29] interesting pod/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 04:27:03.390342 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:03.390229    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t" podUID="30b33bf9-b5cb-473c-b6a4-d30dd097fd72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 04:27:04.378619 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:04.378583    4874 patch_prober.go:29] interesting pod/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 04:27:04.379213 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:04.379179    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t" podUID="30b33bf9-b5cb-473c-b6a4-d30dd097fd72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 04:27:05.386262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:05.385603    4874 patch_prober.go:29] interesting pod/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 04:27:05.386262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:05.385661    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t" podUID="30b33bf9-b5cb-473c-b6a4-d30dd097fd72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 04:27:06.045226 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:06.045194    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6"]
Oct 30 04:27:06.116604 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:06.116569    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8"]
Oct 30 04:27:06.379739 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:06.379643    4874 patch_prober.go:29] interesting pod/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 04:27:06.380029 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:06.379998    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t" podUID="30b33bf9-b5cb-473c-b6a4-d30dd097fd72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 04:27:07.380625 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:07.380587    4874 patch_prober.go:29] interesting pod/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 04:27:07.381221 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:07.381191    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t" podUID="30b33bf9-b5cb-473c-b6a4-d30dd097fd72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 04:27:08.223452 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:08.223382    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65" podUID="252d95cc-17a2-450d-b64f-29a1c0b48136" containerName="payload-pre-processing" containerID="cri-o://6ec9c06685e38a64b2a217d716770e9badfeded8dfb5739f5175428a380bbf0b" gracePeriod=25
Oct 30 04:27:08.350071 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:08.350036    4874 generic.go:356] "Generic (PLEG): container finished" podID="252d95cc-17a2-450d-b64f-29a1c0b48136" containerID="6ec9c06685e38a64b2a217d716770e9badfeded8dfb5739f5175428a380bbf0b" exitCode=0
Oct 30 04:27:08.350223 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:08.350098    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65" event={"ID":"252d95cc-17a2-450d-b64f-29a1c0b48136","Type":"ContainerDied","Data":"6ec9c06685e38a64b2a217d716770e9badfeded8dfb5739f5175428a380bbf0b"}
Oct 30 04:27:08.456355 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:08.456323    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:27:08.457332 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:08.457278    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:27:08.458176 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:08.458154    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:27:08.497358 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:08.497337    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65"
Oct 30 04:27:08.633487 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:08.633460    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/252d95cc-17a2-450d-b64f-29a1c0b48136-kube-api-access-qbpdd\" (UniqueName: \"kubernetes.io/projected/252d95cc-17a2-450d-b64f-29a1c0b48136-kube-api-access-qbpdd\") pod \"252d95cc-17a2-450d-b64f-29a1c0b48136\" (UID: \"252d95cc-17a2-450d-b64f-29a1c0b48136\") "
Oct 30 04:27:08.633658 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:08.633625    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/252d95cc-17a2-450d-b64f-29a1c0b48136-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/252d95cc-17a2-450d-b64f-29a1c0b48136-plugins-config-volume\") pod \"252d95cc-17a2-450d-b64f-29a1c0b48136\" (UID: \"252d95cc-17a2-450d-b64f-29a1c0b48136\") "
Oct 30 04:27:08.634040 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:08.634007    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/252d95cc-17a2-450d-b64f-29a1c0b48136-plugins-config-volume" pod "252d95cc-17a2-450d-b64f-29a1c0b48136" (UID: "252d95cc-17a2-450d-b64f-29a1c0b48136"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:27:08.634291 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:08.634275    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/252d95cc-17a2-450d-b64f-29a1c0b48136-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:27:08.635631 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:08.635610    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/252d95cc-17a2-450d-b64f-29a1c0b48136-kube-api-access-qbpdd" pod "252d95cc-17a2-450d-b64f-29a1c0b48136" (UID: "252d95cc-17a2-450d-b64f-29a1c0b48136"). InnerVolumeSpecName "kube-api-access-qbpdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:27:08.735104 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:08.735034    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbpdd\" (UniqueName: \"kubernetes.io/projected/252d95cc-17a2-450d-b64f-29a1c0b48136-kube-api-access-qbpdd\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:27:09.358574 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:09.358546    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65"
Oct 30 04:27:09.358752 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:09.358586    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65" event={"ID":"252d95cc-17a2-450d-b64f-29a1c0b48136","Type":"ContainerDied","Data":"b18d08bc68b9afaced42c054eb378155511fd738cb45540805c6f05d96d1ea9a"}
Oct 30 04:27:09.358752 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:09.358646    4874 scope.go:122] "RemoveContainer" containerID="6ec9c06685e38a64b2a217d716770e9badfeded8dfb5739f5175428a380bbf0b"
Oct 30 04:27:09.408463 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:09.408429    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65"]
Oct 30 04:27:09.419448 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:09.419423    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-baa474-cb9b87fdc-xhf65"]
Oct 30 04:27:10.729425 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:10.729392    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="252d95cc-17a2-450d-b64f-29a1c0b48136" path="/var/lib/kubelet/pods/252d95cc-17a2-450d-b64f-29a1c0b48136/volumes"
Oct 30 04:27:11.057321 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:11.056346    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6" podUID="bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4" containerName="payload-processing" containerID="cri-o://2bbf3826e873814a098981de7a1866f22268850a523be204bd1f9bbd858c598a" gracePeriod=25
Oct 30 04:27:11.119541 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:11.119316    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8" podUID="37adc92c-e067-479f-b982-c03987b298a2" containerName="payload-pre-processing" containerID="cri-o://6b62fb2f5dd91ba3ab2636b015dbfd2bed625340d1895c65d7b0fd13e45790e1" gracePeriod=25
Oct 30 04:27:11.380264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:11.380204    4874 generic.go:356] "Generic (PLEG): container finished" podID="bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4" containerID="2bbf3826e873814a098981de7a1866f22268850a523be204bd1f9bbd858c598a" exitCode=0
Oct 30 04:27:11.380404 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:11.380277    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6" event={"ID":"bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4","Type":"ContainerDied","Data":"2bbf3826e873814a098981de7a1866f22268850a523be204bd1f9bbd858c598a"}
Oct 30 04:27:11.382076 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:11.382048    4874 generic.go:356] "Generic (PLEG): container finished" podID="37adc92c-e067-479f-b982-c03987b298a2" containerID="6b62fb2f5dd91ba3ab2636b015dbfd2bed625340d1895c65d7b0fd13e45790e1" exitCode=0
Oct 30 04:27:11.382219 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:11.382104    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8" event={"ID":"37adc92c-e067-479f-b982-c03987b298a2","Type":"ContainerDied","Data":"6b62fb2f5dd91ba3ab2636b015dbfd2bed625340d1895c65d7b0fd13e45790e1"}
Oct 30 04:27:11.427799 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:11.427771    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6"
Oct 30 04:27:11.460169 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:11.460146    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8"
Oct 30 04:27:11.488400 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:11.488366    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4-plugins-config-volume\") pod \"bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4\" (UID: \"bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4\") "
Oct 30 04:27:11.488400 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:11.488402    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37adc92c-e067-479f-b982-c03987b298a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37adc92c-e067-479f-b982-c03987b298a2-plugins-config-volume\") pod \"37adc92c-e067-479f-b982-c03987b298a2\" (UID: \"37adc92c-e067-479f-b982-c03987b298a2\") "
Oct 30 04:27:11.488631 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:11.488480    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4-kube-api-access-sjwqk\" (UniqueName: \"kubernetes.io/projected/bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4-kube-api-access-sjwqk\") pod \"bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4\" (UID: \"bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4\") "
Oct 30 04:27:11.488631 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:11.488592    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37adc92c-e067-479f-b982-c03987b298a2-kube-api-access-htsr8\" (UniqueName: \"kubernetes.io/projected/37adc92c-e067-479f-b982-c03987b298a2-kube-api-access-htsr8\") pod \"37adc92c-e067-479f-b982-c03987b298a2\" (UID: \"37adc92c-e067-479f-b982-c03987b298a2\") "
Oct 30 04:27:11.488846 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:11.488816    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4-plugins-config-volume" pod "bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4" (UID: "bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:27:11.489047 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:11.489024    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:27:11.489616 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:11.489586    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37adc92c-e067-479f-b982-c03987b298a2-plugins-config-volume" pod "37adc92c-e067-479f-b982-c03987b298a2" (UID: "37adc92c-e067-479f-b982-c03987b298a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:27:11.491592 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:11.491559    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4-kube-api-access-sjwqk" pod "bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4" (UID: "bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4"). InnerVolumeSpecName "kube-api-access-sjwqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:27:11.491698 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:11.491627    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37adc92c-e067-479f-b982-c03987b298a2-kube-api-access-htsr8" pod "37adc92c-e067-479f-b982-c03987b298a2" (UID: "37adc92c-e067-479f-b982-c03987b298a2"). InnerVolumeSpecName "kube-api-access-htsr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:27:11.595124 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:11.591297    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37adc92c-e067-479f-b982-c03987b298a2-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:27:11.595124 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:11.591333    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjwqk\" (UniqueName: \"kubernetes.io/projected/bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4-kube-api-access-sjwqk\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:27:11.595124 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:11.591349    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htsr8\" (UniqueName: \"kubernetes.io/projected/37adc92c-e067-479f-b982-c03987b298a2-kube-api-access-htsr8\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:27:12.389440 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:12.389360    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6"
Oct 30 04:27:12.389440 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:12.389383    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6" event={"ID":"bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4","Type":"ContainerDied","Data":"303884e661fa2ace6f018ce907ee51dd7168df5d0bd352fa0ce58526b002415c"}
Oct 30 04:27:12.389943 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:12.389459    4874 scope.go:122] "RemoveContainer" containerID="2bbf3826e873814a098981de7a1866f22268850a523be204bd1f9bbd858c598a"
Oct 30 04:27:12.390874 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:12.390853    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8" event={"ID":"37adc92c-e067-479f-b982-c03987b298a2","Type":"ContainerDied","Data":"b7a7d84d3db3f4dcc546e7843487b2922f83052cd9af3ce0ceab0d876a58dbe4"}
Oct 30 04:27:12.390960 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:12.390899    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8"
Oct 30 04:27:12.399748 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:12.399728    4874 scope.go:122] "RemoveContainer" containerID="6b62fb2f5dd91ba3ab2636b015dbfd2bed625340d1895c65d7b0fd13e45790e1"
Oct 30 04:27:12.439489 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:12.439457    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8"]
Oct 30 04:27:12.447382 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:12.447358    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1a0d3427-67dd99d479-zmcx8"]
Oct 30 04:27:12.480027 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:12.479995    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6"]
Oct 30 04:27:12.496270 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:12.496234    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-bbrl6"]
Oct 30 04:27:12.772270 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:12.770497    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37adc92c-e067-479f-b982-c03987b298a2" path="/var/lib/kubelet/pods/37adc92c-e067-479f-b982-c03987b298a2/volumes"
Oct 30 04:27:12.772270 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:12.771066    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4" path="/var/lib/kubelet/pods/bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4/volumes"
Oct 30 04:27:17.545541 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:17.545469    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f"]
Oct 30 04:27:17.545980 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:17.545967    4874 cpu_manager.go:414] "containerMap: removing container" podUID="bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4" containerName="payload-processing"
Oct 30 04:27:17.546023 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:17.545981    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4" containerName="payload-processing"
Oct 30 04:27:17.546023 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:17.545990    4874 cpu_manager.go:414] "containerMap: removing container" podUID="252d95cc-17a2-450d-b64f-29a1c0b48136" containerName="payload-pre-processing"
Oct 30 04:27:17.546023 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:17.545995    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="252d95cc-17a2-450d-b64f-29a1c0b48136" containerName="payload-pre-processing"
Oct 30 04:27:17.546023 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:17.546005    4874 cpu_manager.go:414] "containerMap: removing container" podUID="37adc92c-e067-479f-b982-c03987b298a2" containerName="payload-pre-processing"
Oct 30 04:27:17.546023 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:17.546009    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37adc92c-e067-479f-b982-c03987b298a2" containerName="payload-pre-processing"
Oct 30 04:27:17.546182 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:17.546079    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7f3ecc-4de5-4bb3-ab57-596f594c1bc4" containerName="payload-processing"
Oct 30 04:27:17.546182 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:17.546087    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="252d95cc-17a2-450d-b64f-29a1c0b48136" containerName="payload-pre-processing"
Oct 30 04:27:17.546182 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:17.546094    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="37adc92c-e067-479f-b982-c03987b298a2" containerName="payload-pre-processing"
Oct 30 04:27:17.548833 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:17.548815    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f"
Oct 30 04:27:17.562754 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:17.562728    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-baa474\""
Oct 30 04:27:17.574280 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:17.574254    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-baa474-dockercfg-dgwzk\""
Oct 30 04:27:17.576745 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:17.576723    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f"]
Oct 30 04:27:17.695167 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:17.695138    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83b9efdd-4a34-428e-9f19-3ad6be585fe0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f\" (UID: \"83b9efdd-4a34-428e-9f19-3ad6be585fe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f"
Oct 30 04:27:17.695359 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:17.695205    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm99z\" (UniqueName: \"kubernetes.io/projected/83b9efdd-4a34-428e-9f19-3ad6be585fe0-kube-api-access-xm99z\") pod \"payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f\" (UID: \"83b9efdd-4a34-428e-9f19-3ad6be585fe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f"
Oct 30 04:27:17.796493 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:17.796406    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm99z\" (UniqueName: \"kubernetes.io/projected/83b9efdd-4a34-428e-9f19-3ad6be585fe0-kube-api-access-xm99z\") pod \"payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f\" (UID: \"83b9efdd-4a34-428e-9f19-3ad6be585fe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f"
Oct 30 04:27:17.796650 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:17.796547    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83b9efdd-4a34-428e-9f19-3ad6be585fe0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f\" (UID: \"83b9efdd-4a34-428e-9f19-3ad6be585fe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f"
Oct 30 04:27:17.797225 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:17.797203    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83b9efdd-4a34-428e-9f19-3ad6be585fe0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f\" (UID: \"83b9efdd-4a34-428e-9f19-3ad6be585fe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f"
Oct 30 04:27:17.819637 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:17.819610    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm99z\" (UniqueName: \"kubernetes.io/projected/83b9efdd-4a34-428e-9f19-3ad6be585fe0-kube-api-access-xm99z\") pod \"payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f\" (UID: \"83b9efdd-4a34-428e-9f19-3ad6be585fe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f"
Oct 30 04:27:17.861441 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:17.861416    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f"
Oct 30 04:27:18.183313 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:18.183169    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f"]
Oct 30 04:27:18.213027 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:27:18.212989    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b9efdd_4a34_428e_9f19_3ad6be585fe0.slice/crio-466e07587cfcd923bf15c4ee596cbb2830c6966dfdf0f42c7669fab3fd6250f7 WatchSource:0}: Error finding container 466e07587cfcd923bf15c4ee596cbb2830c6966dfdf0f42c7669fab3fd6250f7: Status 404 returned error can't find the container with id 466e07587cfcd923bf15c4ee596cbb2830c6966dfdf0f42c7669fab3fd6250f7
Oct 30 04:27:18.426477 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:18.426447    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f" event={"ID":"83b9efdd-4a34-428e-9f19-3ad6be585fe0","Type":"ContainerStarted","Data":"466e07587cfcd923bf15c4ee596cbb2830c6966dfdf0f42c7669fab3fd6250f7"}
Oct 30 04:27:20.444146 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:20.444104    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f" event={"ID":"83b9efdd-4a34-428e-9f19-3ad6be585fe0","Type":"ContainerStarted","Data":"55aa0b0bd26227f078b51eaad0ce87dedc9bb0ef982bf092e3b4947f146ddeae"}
Oct 30 04:27:20.444557 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:20.444331    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f"
Oct 30 04:27:20.496308 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:20.496260    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f" podStartSLOduration=2.090151748 podStartE2EDuration="3.496247353s" totalImagesPullingTime="1.406095605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:27:20.493263427 +0000 UTC m=+1759.905659001" watchObservedRunningTime="2026-10-30 04:27:20.496247353 +0000 UTC m=+1759.908642917"
Oct 30 04:27:21.730972 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:21.730937    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6"]
Oct 30 04:27:21.731504 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:21.731163    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6" podUID="da1b9a2c-09c4-4115-b48d-8d6ae387561c" containerName="maas-api" containerID="cri-o://81d1b2bcb372683d9840492327082ca9dd6dd716295587d6c04d2af9283809c9" gracePeriod=30
Oct 30 04:27:22.043090 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:22.043055    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6"
Oct 30 04:27:22.218264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:22.180426    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da1b9a2c-09c4-4115-b48d-8d6ae387561c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da1b9a2c-09c4-4115-b48d-8d6ae387561c-maas-api-tls\") pod \"da1b9a2c-09c4-4115-b48d-8d6ae387561c\" (UID: \"da1b9a2c-09c4-4115-b48d-8d6ae387561c\") "
Oct 30 04:27:22.218264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:22.180579    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da1b9a2c-09c4-4115-b48d-8d6ae387561c-kube-api-access-wbhd7\" (UniqueName: \"kubernetes.io/projected/da1b9a2c-09c4-4115-b48d-8d6ae387561c-kube-api-access-wbhd7\") pod \"da1b9a2c-09c4-4115-b48d-8d6ae387561c\" (UID: \"da1b9a2c-09c4-4115-b48d-8d6ae387561c\") "
Oct 30 04:27:22.218264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:22.180707    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da1b9a2c-09c4-4115-b48d-8d6ae387561c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da1b9a2c-09c4-4115-b48d-8d6ae387561c-metrics-certs\") pod \"da1b9a2c-09c4-4115-b48d-8d6ae387561c\" (UID: \"da1b9a2c-09c4-4115-b48d-8d6ae387561c\") "
Oct 30 04:27:22.218264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:22.180758    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da1b9a2c-09c4-4115-b48d-8d6ae387561c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da1b9a2c-09c4-4115-b48d-8d6ae387561c-openshift-service-ca\") pod \"da1b9a2c-09c4-4115-b48d-8d6ae387561c\" (UID: \"da1b9a2c-09c4-4115-b48d-8d6ae387561c\") "
Oct 30 04:27:22.218264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:22.202469    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da1b9a2c-09c4-4115-b48d-8d6ae387561c-maas-api-tls" pod "da1b9a2c-09c4-4115-b48d-8d6ae387561c" (UID: "da1b9a2c-09c4-4115-b48d-8d6ae387561c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:27:22.218264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:22.202548    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da1b9a2c-09c4-4115-b48d-8d6ae387561c-metrics-certs" pod "da1b9a2c-09c4-4115-b48d-8d6ae387561c" (UID: "da1b9a2c-09c4-4115-b48d-8d6ae387561c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:27:22.218264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:22.202625    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1b9a2c-09c4-4115-b48d-8d6ae387561c-kube-api-access-wbhd7" pod "da1b9a2c-09c4-4115-b48d-8d6ae387561c" (UID: "da1b9a2c-09c4-4115-b48d-8d6ae387561c"). InnerVolumeSpecName "kube-api-access-wbhd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:27:22.218874 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:22.218442    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da1b9a2c-09c4-4115-b48d-8d6ae387561c-openshift-service-ca" pod "da1b9a2c-09c4-4115-b48d-8d6ae387561c" (UID: "da1b9a2c-09c4-4115-b48d-8d6ae387561c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:27:22.286979 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:22.286892    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da1b9a2c-09c4-4115-b48d-8d6ae387561c-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:27:22.286979 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:22.286923    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da1b9a2c-09c4-4115-b48d-8d6ae387561c-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:27:22.286979 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:22.286936    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da1b9a2c-09c4-4115-b48d-8d6ae387561c-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:27:22.286979 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:22.286950    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbhd7\" (UniqueName: \"kubernetes.io/projected/da1b9a2c-09c4-4115-b48d-8d6ae387561c-kube-api-access-wbhd7\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:27:22.456992 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:22.456952    4874 generic.go:356] "Generic (PLEG): container finished" podID="da1b9a2c-09c4-4115-b48d-8d6ae387561c" containerID="81d1b2bcb372683d9840492327082ca9dd6dd716295587d6c04d2af9283809c9" exitCode=0
Oct 30 04:27:22.457457 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:22.457430    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6" event={"ID":"da1b9a2c-09c4-4115-b48d-8d6ae387561c","Type":"ContainerDied","Data":"81d1b2bcb372683d9840492327082ca9dd6dd716295587d6c04d2af9283809c9"}
Oct 30 04:27:22.457747 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:22.457712    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6" event={"ID":"da1b9a2c-09c4-4115-b48d-8d6ae387561c","Type":"ContainerDied","Data":"df8a26dff4908229204acee837aca47f889e249b40a2ea96099fdaf509fdb7dc"}
Oct 30 04:27:22.457982 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:22.457967    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6"
Oct 30 04:27:22.459061 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:22.459039    4874 scope.go:122] "RemoveContainer" containerID="81d1b2bcb372683d9840492327082ca9dd6dd716295587d6c04d2af9283809c9"
Oct 30 04:27:22.502588 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:22.502468    4874 scope.go:122] "RemoveContainer" containerID="81d1b2bcb372683d9840492327082ca9dd6dd716295587d6c04d2af9283809c9"
Oct 30 04:27:22.503056 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:27:22.503029    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81d1b2bcb372683d9840492327082ca9dd6dd716295587d6c04d2af9283809c9\": container with ID starting with 81d1b2bcb372683d9840492327082ca9dd6dd716295587d6c04d2af9283809c9 not found: ID does not exist" containerID="81d1b2bcb372683d9840492327082ca9dd6dd716295587d6c04d2af9283809c9"
Oct 30 04:27:22.503190 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:22.503065    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d1b2bcb372683d9840492327082ca9dd6dd716295587d6c04d2af9283809c9"} err="failed to get container status \"81d1b2bcb372683d9840492327082ca9dd6dd716295587d6c04d2af9283809c9\": rpc error: code = NotFound desc = could not find container \"81d1b2bcb372683d9840492327082ca9dd6dd716295587d6c04d2af9283809c9\": container with ID starting with 81d1b2bcb372683d9840492327082ca9dd6dd716295587d6c04d2af9283809c9 not found: ID does not exist"
Oct 30 04:27:22.546212 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:22.546132    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6"]
Oct 30 04:27:22.552519 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:22.552492    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1a0d3427-6bc4fd546-grdk6"]
Oct 30 04:27:22.729078 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:22.729045    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da1b9a2c-09c4-4115-b48d-8d6ae387561c" path="/var/lib/kubelet/pods/da1b9a2c-09c4-4115-b48d-8d6ae387561c/volumes"
Oct 30 04:27:23.997762 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:23.997724    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:27:25.609346 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:27:25.609309    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:27:26.292496 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.292463    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"]
Oct 30 04:27:26.293042 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.293023    4874 cpu_manager.go:414] "containerMap: removing container" podUID="da1b9a2c-09c4-4115-b48d-8d6ae387561c" containerName="maas-api"
Oct 30 04:27:26.293042 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.293043    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da1b9a2c-09c4-4115-b48d-8d6ae387561c" containerName="maas-api"
Oct 30 04:27:26.293215 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.293195    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="da1b9a2c-09c4-4115-b48d-8d6ae387561c" containerName="maas-api"
Oct 30 04:27:26.295569 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.295548    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.322946 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.322913    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"]
Oct 30 04:27:26.329744 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.329715    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-envoy\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.329883 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.329838    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-workload-socket\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.329975 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.329947    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-podinfo\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.330042 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.330027    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-credential-socket\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.330106 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.330066    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istiod-ca-cert\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.330106 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.330098    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-token\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.330219 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.330174    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-workload-certs\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.330303 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.330285    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-data\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.330363 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.330337    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ktnk\" (UniqueName: \"kubernetes.io/projected/ba8e0c94-8f6b-40dd-86f9-3973177dd696-kube-api-access-5ktnk\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.330863 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.330839    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b1a417-openshift-default-dockercfg-rmbbb\""
Oct 30 04:27:26.431554 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.431522    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-envoy\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.431731 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.431582    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-workload-socket\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.431731 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.431626    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-podinfo\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.431731 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.431681    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-credential-socket\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.431731 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.431714    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istiod-ca-cert\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.431969 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.431740    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-token\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.431969 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.431769    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-workload-certs\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.431969 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.431855    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-data\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.431969 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.431880    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ktnk\" (UniqueName: \"kubernetes.io/projected/ba8e0c94-8f6b-40dd-86f9-3973177dd696-kube-api-access-5ktnk\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.432191 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.432067    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-workload-socket\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.432191 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.432100    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-credential-socket\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.432191 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.432137    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-workload-certs\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.432364 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.432343    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-data\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.432662 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.432644    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istiod-ca-cert\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.433837 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.433817    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-envoy\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.434070 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.434052    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-podinfo\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.582561 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.581769    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-token\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.593949 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.591933    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ktnk\" (UniqueName: \"kubernetes.io/projected/ba8e0c94-8f6b-40dd-86f9-3973177dd696-kube-api-access-5ktnk\") pod \"e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") " pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.610767 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.610476    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:26.938430 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.938275    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"]
Oct 30 04:27:26.941191 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:27:26.941156    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8e0c94_8f6b_40dd_86f9_3973177dd696.slice/crio-a379274e28679c2a438590e7a6aab93bc80ef5721cdee61d37d23bb543ae4154 WatchSource:0}: Error finding container a379274e28679c2a438590e7a6aab93bc80ef5721cdee61d37d23bb543ae4154: Status 404 returned error can't find the container with id a379274e28679c2a438590e7a6aab93bc80ef5721cdee61d37d23bb543ae4154
Oct 30 04:27:26.943545 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.943515    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:27:26.943618 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.943578    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:27:26.943618 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:26.943605    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:27:27.546133 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:27.546094    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz" event={"ID":"ba8e0c94-8f6b-40dd-86f9-3973177dd696","Type":"ContainerStarted","Data":"ed12f2000ad6f93266022a3c200402a5513443b3ed59dde9737d68edfe9eff4e"}
Oct 30 04:27:27.546334 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:27.546158    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz" event={"ID":"ba8e0c94-8f6b-40dd-86f9-3973177dd696","Type":"ContainerStarted","Data":"a379274e28679c2a438590e7a6aab93bc80ef5721cdee61d37d23bb543ae4154"}
Oct 30 04:27:27.611474 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:27.611446    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:27.617339 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:27.617279    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz" podStartSLOduration=1.6172619510000001 podStartE2EDuration="1.617261951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:27:27.604032953 +0000 UTC m=+1767.016428525" watchObservedRunningTime="2026-10-30 04:27:27.617261951 +0000 UTC m=+1767.029657526"
Oct 30 04:27:27.617833 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:27.617809    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:28.552228 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:28.552200    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:28.553005 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:28.552982    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:27:31.450815 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:31.450782    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f"
Oct 30 04:27:31.832824 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:31.832794    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc"]
Oct 30 04:27:31.835650 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:31.835628    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc"
Oct 30 04:27:31.860797 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:31.860765    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b1a417\""
Oct 30 04:27:31.861111 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:31.861090    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc"]
Oct 30 04:27:31.863388 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:31.863367    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b1a417-dockercfg-6mhx4\""
Oct 30 04:27:31.942505 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:31.942468    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq9hb\" (UniqueName: \"kubernetes.io/projected/aab79284-1304-4ade-8377-6ea389a3b63a-kube-api-access-mq9hb\") pod \"payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc\" (UID: \"aab79284-1304-4ade-8377-6ea389a3b63a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc"
Oct 30 04:27:31.942887 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:31.942862    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aab79284-1304-4ade-8377-6ea389a3b63a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc\" (UID: \"aab79284-1304-4ade-8377-6ea389a3b63a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc"
Oct 30 04:27:32.047295 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:32.047136    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq9hb\" (UniqueName: \"kubernetes.io/projected/aab79284-1304-4ade-8377-6ea389a3b63a-kube-api-access-mq9hb\") pod \"payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc\" (UID: \"aab79284-1304-4ade-8377-6ea389a3b63a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc"
Oct 30 04:27:32.047496 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:32.047303    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aab79284-1304-4ade-8377-6ea389a3b63a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc\" (UID: \"aab79284-1304-4ade-8377-6ea389a3b63a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc"
Oct 30 04:27:32.048353 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:32.048283    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aab79284-1304-4ade-8377-6ea389a3b63a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc\" (UID: \"aab79284-1304-4ade-8377-6ea389a3b63a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc"
Oct 30 04:27:32.075977 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:32.075944    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq9hb\" (UniqueName: \"kubernetes.io/projected/aab79284-1304-4ade-8377-6ea389a3b63a-kube-api-access-mq9hb\") pod \"payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc\" (UID: \"aab79284-1304-4ade-8377-6ea389a3b63a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc"
Oct 30 04:27:32.148430 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:32.148347    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc"
Oct 30 04:27:32.513372 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:32.513121    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc"]
Oct 30 04:27:32.515973 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:27:32.515947    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab79284_1304_4ade_8377_6ea389a3b63a.slice/crio-978e29ea592545511b4c3b6500b95984fbc1c71f2eb219727417ca20cee4f305 WatchSource:0}: Error finding container 978e29ea592545511b4c3b6500b95984fbc1c71f2eb219727417ca20cee4f305: Status 404 returned error can't find the container with id 978e29ea592545511b4c3b6500b95984fbc1c71f2eb219727417ca20cee4f305
Oct 30 04:27:32.577815 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:32.577789    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc" event={"ID":"aab79284-1304-4ade-8377-6ea389a3b63a","Type":"ContainerStarted","Data":"978e29ea592545511b4c3b6500b95984fbc1c71f2eb219727417ca20cee4f305"}
Oct 30 04:27:33.584964 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:33.584931    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc" event={"ID":"aab79284-1304-4ade-8377-6ea389a3b63a","Type":"ContainerStarted","Data":"7edeb00f1cff4651970c0fb0eaf8e005b9a116903c61bb83ca5549327961c67b"}
Oct 30 04:27:33.585391 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:33.585067    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc"
Oct 30 04:27:33.627304 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:33.627226    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc" podStartSLOduration=2.627208501 podStartE2EDuration="2.627208501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:27:33.623642097 +0000 UTC m=+1773.036037671" watchObservedRunningTime="2026-10-30 04:27:33.627208501 +0000 UTC m=+1773.039604074"
Oct 30 04:27:41.594416 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:41.594387    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf"]
Oct 30 04:27:41.598545 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:41.598525    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf"
Oct 30 04:27:41.619658 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:41.619634    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf"]
Oct 30 04:27:41.624876 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:41.624850    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-763bde\""
Oct 30 04:27:41.625026 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:41.624899    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-763bde-dockercfg-rjnlk\""
Oct 30 04:27:41.716263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:41.691095    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzs5x\" (UniqueName: \"kubernetes.io/projected/8682db73-4941-40f5-88b6-54c04bc2212f-kube-api-access-pzs5x\") pod \"payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf\" (UID: \"8682db73-4941-40f5-88b6-54c04bc2212f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf"
Oct 30 04:27:41.716263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:41.691311    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8682db73-4941-40f5-88b6-54c04bc2212f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf\" (UID: \"8682db73-4941-40f5-88b6-54c04bc2212f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf"
Oct 30 04:27:41.718095 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:41.718066    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r"]
Oct 30 04:27:41.731262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:41.722956    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r"
Oct 30 04:27:41.773689 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:41.758810    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r"]
Oct 30 04:27:41.794157 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:41.794124    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzs5x\" (UniqueName: \"kubernetes.io/projected/8682db73-4941-40f5-88b6-54c04bc2212f-kube-api-access-pzs5x\") pod \"payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf\" (UID: \"8682db73-4941-40f5-88b6-54c04bc2212f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf"
Oct 30 04:27:41.794624 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:41.794595    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8682db73-4941-40f5-88b6-54c04bc2212f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf\" (UID: \"8682db73-4941-40f5-88b6-54c04bc2212f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf"
Oct 30 04:27:41.796509 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:41.796481    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8682db73-4941-40f5-88b6-54c04bc2212f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf\" (UID: \"8682db73-4941-40f5-88b6-54c04bc2212f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf"
Oct 30 04:27:41.820277 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:41.820223    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzs5x\" (UniqueName: \"kubernetes.io/projected/8682db73-4941-40f5-88b6-54c04bc2212f-kube-api-access-pzs5x\") pod \"payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf\" (UID: \"8682db73-4941-40f5-88b6-54c04bc2212f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf"
Oct 30 04:27:41.895807 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:41.895738    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/572594dd-6b8e-419f-a838-62671f6426c3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r\" (UID: \"572594dd-6b8e-419f-a838-62671f6426c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r"
Oct 30 04:27:41.895935 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:41.895865    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcfsb\" (UniqueName: \"kubernetes.io/projected/572594dd-6b8e-419f-a838-62671f6426c3-kube-api-access-dcfsb\") pod \"payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r\" (UID: \"572594dd-6b8e-419f-a838-62671f6426c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r"
Oct 30 04:27:41.911496 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:41.911469    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf"
Oct 30 04:27:41.997142 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:41.997110    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/572594dd-6b8e-419f-a838-62671f6426c3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r\" (UID: \"572594dd-6b8e-419f-a838-62671f6426c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r"
Oct 30 04:27:41.997341 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:41.997234    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcfsb\" (UniqueName: \"kubernetes.io/projected/572594dd-6b8e-419f-a838-62671f6426c3-kube-api-access-dcfsb\") pod \"payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r\" (UID: \"572594dd-6b8e-419f-a838-62671f6426c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r"
Oct 30 04:27:41.997842 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:41.997823    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/572594dd-6b8e-419f-a838-62671f6426c3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r\" (UID: \"572594dd-6b8e-419f-a838-62671f6426c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r"
Oct 30 04:27:42.020838 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:42.020811    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcfsb\" (UniqueName: \"kubernetes.io/projected/572594dd-6b8e-419f-a838-62671f6426c3-kube-api-access-dcfsb\") pod \"payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r\" (UID: \"572594dd-6b8e-419f-a838-62671f6426c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r"
Oct 30 04:27:42.082907 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:42.082878    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r"
Oct 30 04:27:42.230671 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:42.230650    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf"]
Oct 30 04:27:42.232794 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:27:42.232768    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8682db73_4941_40f5_88b6_54c04bc2212f.slice/crio-698364a197cd5724a8d98248fb766652398ee985b9e11569b9772ab95bb36147 WatchSource:0}: Error finding container 698364a197cd5724a8d98248fb766652398ee985b9e11569b9772ab95bb36147: Status 404 returned error can't find the container with id 698364a197cd5724a8d98248fb766652398ee985b9e11569b9772ab95bb36147
Oct 30 04:27:42.402358 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:42.402282    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r"]
Oct 30 04:27:42.403706 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:27:42.403676    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod572594dd_6b8e_419f_a838_62671f6426c3.slice/crio-11a735fb41e3cb660647f6112b99f796d9b97a3a6066d79a90d7365899195687 WatchSource:0}: Error finding container 11a735fb41e3cb660647f6112b99f796d9b97a3a6066d79a90d7365899195687: Status 404 returned error can't find the container with id 11a735fb41e3cb660647f6112b99f796d9b97a3a6066d79a90d7365899195687
Oct 30 04:27:42.636361 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:42.636265    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r" event={"ID":"572594dd-6b8e-419f-a838-62671f6426c3","Type":"ContainerStarted","Data":"1ef168ee1007cfee300cfb0722620268d22b636efb7ba687f5379cd622486510"}
Oct 30 04:27:42.636361 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:42.636314    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r" event={"ID":"572594dd-6b8e-419f-a838-62671f6426c3","Type":"ContainerStarted","Data":"11a735fb41e3cb660647f6112b99f796d9b97a3a6066d79a90d7365899195687"}
Oct 30 04:27:42.636801 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:42.636399    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r"
Oct 30 04:27:42.637872 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:42.637851    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf" event={"ID":"8682db73-4941-40f5-88b6-54c04bc2212f","Type":"ContainerStarted","Data":"c37279ff296560eeb020c786088541ed0f94ef344990aaa71fbce7061acfa3cc"}
Oct 30 04:27:42.637993 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:42.637886    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf" event={"ID":"8682db73-4941-40f5-88b6-54c04bc2212f","Type":"ContainerStarted","Data":"698364a197cd5724a8d98248fb766652398ee985b9e11569b9772ab95bb36147"}
Oct 30 04:27:42.637993 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:42.637986    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf"
Oct 30 04:27:42.675804 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:42.675747    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r" podStartSLOduration=1.675732523 podStartE2EDuration="1.675732523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:27:42.674653019 +0000 UTC m=+1782.087048591" watchObservedRunningTime="2026-10-30 04:27:42.675732523 +0000 UTC m=+1782.088128097"
Oct 30 04:27:42.715342 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:42.715291    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf" podStartSLOduration=1.715275571 podStartE2EDuration="1.715275571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:27:42.712927886 +0000 UTC m=+1782.125323458" watchObservedRunningTime="2026-10-30 04:27:42.715275571 +0000 UTC m=+1782.127671146"
Oct 30 04:27:44.592608 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:44.592580    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc"
Oct 30 04:27:47.960390 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:47.960359    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:27:53.646560 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:53.646528    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf"
Oct 30 04:27:53.647033 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:27:53.646617    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r"
Oct 30 04:27:55.646288 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:27:55.646223    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:28:00.890494 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:00.890469    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 04:28:00.921413 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:00.921387    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 04:28:01.565702 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:01.565670    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2"]
Oct 30 04:28:01.571132 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:01.571115    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2"
Oct 30 04:28:01.587087 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:01.587065    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-763bde-serving-cert\""
Oct 30 04:28:01.601280 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:01.601214    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2"]
Oct 30 04:28:01.705765 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:01.705732    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5ml7\" (UniqueName: \"kubernetes.io/projected/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-kube-api-access-v5ml7\") pod \"maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2\" (UID: \"d1950c75-bacc-4f16-91f0-66e2f6b3cf18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2"
Oct 30 04:28:01.705923 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:01.705786    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-metrics-certs\") pod \"maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2\" (UID: \"d1950c75-bacc-4f16-91f0-66e2f6b3cf18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2"
Oct 30 04:28:01.705968 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:01.705915    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-maas-api-tls\") pod \"maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2\" (UID: \"d1950c75-bacc-4f16-91f0-66e2f6b3cf18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2"
Oct 30 04:28:01.705968 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:01.705952    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2\" (UID: \"d1950c75-bacc-4f16-91f0-66e2f6b3cf18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2"
Oct 30 04:28:01.811265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:01.808213    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-maas-api-tls\") pod \"maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2\" (UID: \"d1950c75-bacc-4f16-91f0-66e2f6b3cf18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2"
Oct 30 04:28:01.811265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:01.808316    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2\" (UID: \"d1950c75-bacc-4f16-91f0-66e2f6b3cf18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2"
Oct 30 04:28:01.811265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:01.808494    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5ml7\" (UniqueName: \"kubernetes.io/projected/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-kube-api-access-v5ml7\") pod \"maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2\" (UID: \"d1950c75-bacc-4f16-91f0-66e2f6b3cf18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2"
Oct 30 04:28:01.811265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:01.808559    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-metrics-certs\") pod \"maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2\" (UID: \"d1950c75-bacc-4f16-91f0-66e2f6b3cf18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2"
Oct 30 04:28:01.811265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:01.809880    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2\" (UID: \"d1950c75-bacc-4f16-91f0-66e2f6b3cf18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2"
Oct 30 04:28:01.823837 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:01.823767    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-maas-api-tls\") pod \"maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2\" (UID: \"d1950c75-bacc-4f16-91f0-66e2f6b3cf18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2"
Oct 30 04:28:01.824848 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:01.824823    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-metrics-certs\") pod \"maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2\" (UID: \"d1950c75-bacc-4f16-91f0-66e2f6b3cf18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2"
Oct 30 04:28:01.851274 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:01.851189    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5ml7\" (UniqueName: \"kubernetes.io/projected/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-kube-api-access-v5ml7\") pod \"maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2\" (UID: \"d1950c75-bacc-4f16-91f0-66e2f6b3cf18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2"
Oct 30 04:28:01.883443 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:01.883414    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2"
Oct 30 04:28:02.187128 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:02.187103    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2"]
Oct 30 04:28:02.189235 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:28:02.189201    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1950c75_bacc_4f16_91f0_66e2f6b3cf18.slice/crio-63d849418b929f375e0be3433c16961a4f84f6626419a6427ab225a70dc29af3 WatchSource:0}: Error finding container 63d849418b929f375e0be3433c16961a4f84f6626419a6427ab225a70dc29af3: Status 404 returned error can't find the container with id 63d849418b929f375e0be3433c16961a4f84f6626419a6427ab225a70dc29af3
Oct 30 04:28:03.049775 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:03.049743    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2" event={"ID":"d1950c75-bacc-4f16-91f0-66e2f6b3cf18","Type":"ContainerStarted","Data":"cea7bc32848dcb9f0ee5de150302cf6407d373338ff7661747152d052bba5c21"}
Oct 30 04:28:03.049979 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:03.049791    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2" event={"ID":"d1950c75-bacc-4f16-91f0-66e2f6b3cf18","Type":"ContainerStarted","Data":"63d849418b929f375e0be3433c16961a4f84f6626419a6427ab225a70dc29af3"}
Oct 30 04:28:03.049979 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:03.049857    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2"
Oct 30 04:28:09.063215 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:09.063186    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2"
Oct 30 04:28:09.113180 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:09.113131    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2" podStartSLOduration=7.7933635 podStartE2EDuration="8.113116591s" totalImagesPullingTime="319.753091ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:28:03.093770176 +0000 UTC m=+1802.506165752" watchObservedRunningTime="2026-10-30 04:28:09.113116591 +0000 UTC m=+1808.525512165"
Oct 30 04:28:13.130249 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:13.130203    4874 generic.go:356] "Generic (PLEG): container finished" podID="ba8e0c94-8f6b-40dd-86f9-3973177dd696" containerID="ed12f2000ad6f93266022a3c200402a5513443b3ed59dde9737d68edfe9eff4e" exitCode=0
Oct 30 04:28:13.130678 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:13.130272    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz" event={"ID":"ba8e0c94-8f6b-40dd-86f9-3973177dd696","Type":"ContainerDied","Data":"ed12f2000ad6f93266022a3c200402a5513443b3ed59dde9737d68edfe9eff4e"}
Oct 30 04:28:13.130678 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:13.130631    4874 scope.go:122] "RemoveContainer" containerID="ed12f2000ad6f93266022a3c200402a5513443b3ed59dde9737d68edfe9eff4e"
Oct 30 04:28:13.131915 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:13.131890    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:28:13.131980 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:13.131953    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:28:13.132019 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:13.131984    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:28:13.552097 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:13.552067    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:28:14.155264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:14.152180    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz" event={"ID":"ba8e0c94-8f6b-40dd-86f9-3973177dd696","Type":"ContainerStarted","Data":"ab03c3db0059bcc14c41937f07627fce74e16ef0ee6ad0a9df95c583d0405cff"}
Oct 30 04:28:14.610912 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:14.610873    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:28:14.612231 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:14.612206    4874 patch_prober.go:29] interesting pod/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 04:28:14.612354 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:14.612294    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz" podUID="ba8e0c94-8f6b-40dd-86f9-3973177dd696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 04:28:15.611197 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:15.611161    4874 patch_prober.go:29] interesting pod/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 04:28:15.611599 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:15.611218    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz" podUID="ba8e0c94-8f6b-40dd-86f9-3973177dd696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 04:28:15.891484 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:15.891402    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc"]
Oct 30 04:28:16.617666 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:16.617627    4874 patch_prober.go:29] interesting pod/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 04:28:16.618315 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:16.618280    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz" podUID="ba8e0c94-8f6b-40dd-86f9-3973177dd696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 04:28:17.611528 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:17.611488    4874 patch_prober.go:29] interesting pod/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 04:28:17.611787 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:17.611549    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz" podUID="ba8e0c94-8f6b-40dd-86f9-3973177dd696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 04:28:18.582263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:18.581764    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:28:18.614267 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:18.611644    4874 patch_prober.go:29] interesting pod/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 04:28:18.614267 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:18.611720    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz" podUID="ba8e0c94-8f6b-40dd-86f9-3973177dd696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 04:28:19.611152 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:19.611115    4874 patch_prober.go:29] interesting pod/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 04:28:19.611580 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:19.611180    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz" podUID="ba8e0c94-8f6b-40dd-86f9-3973177dd696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 04:28:20.615774 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:20.615742    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:28:20.616140 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:20.615991    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:28:20.616786 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:20.616768    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:28:20.895304 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:20.892331    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc" podUID="aab79284-1304-4ade-8377-6ea389a3b63a" containerName="payload-processing" containerID="cri-o://7edeb00f1cff4651970c0fb0eaf8e005b9a116903c61bb83ca5549327961c67b" gracePeriod=25
Oct 30 04:28:21.159230 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:21.159175    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc"
Oct 30 04:28:21.224446 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:21.224409    4874 generic.go:356] "Generic (PLEG): container finished" podID="aab79284-1304-4ade-8377-6ea389a3b63a" containerID="7edeb00f1cff4651970c0fb0eaf8e005b9a116903c61bb83ca5549327961c67b" exitCode=0
Oct 30 04:28:21.224639 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:21.224458    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc"
Oct 30 04:28:21.224639 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:21.224488    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc" event={"ID":"aab79284-1304-4ade-8377-6ea389a3b63a","Type":"ContainerDied","Data":"7edeb00f1cff4651970c0fb0eaf8e005b9a116903c61bb83ca5549327961c67b"}
Oct 30 04:28:21.224639 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:21.224547    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc" event={"ID":"aab79284-1304-4ade-8377-6ea389a3b63a","Type":"ContainerDied","Data":"978e29ea592545511b4c3b6500b95984fbc1c71f2eb219727417ca20cee4f305"}
Oct 30 04:28:21.224639 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:21.224574    4874 scope.go:122] "RemoveContainer" containerID="7edeb00f1cff4651970c0fb0eaf8e005b9a116903c61bb83ca5549327961c67b"
Oct 30 04:28:21.236909 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:21.236882    4874 scope.go:122] "RemoveContainer" containerID="7edeb00f1cff4651970c0fb0eaf8e005b9a116903c61bb83ca5549327961c67b"
Oct 30 04:28:21.237201 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:28:21.237172    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7edeb00f1cff4651970c0fb0eaf8e005b9a116903c61bb83ca5549327961c67b\": container with ID starting with 7edeb00f1cff4651970c0fb0eaf8e005b9a116903c61bb83ca5549327961c67b not found: ID does not exist" containerID="7edeb00f1cff4651970c0fb0eaf8e005b9a116903c61bb83ca5549327961c67b"
Oct 30 04:28:21.237323 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:21.237209    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7edeb00f1cff4651970c0fb0eaf8e005b9a116903c61bb83ca5549327961c67b"} err="failed to get container status \"7edeb00f1cff4651970c0fb0eaf8e005b9a116903c61bb83ca5549327961c67b\": rpc error: code = NotFound desc = could not find container \"7edeb00f1cff4651970c0fb0eaf8e005b9a116903c61bb83ca5549327961c67b\": container with ID starting with 7edeb00f1cff4651970c0fb0eaf8e005b9a116903c61bb83ca5549327961c67b not found: ID does not exist"
Oct 30 04:28:21.249644 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:21.249622    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aab79284-1304-4ade-8377-6ea389a3b63a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aab79284-1304-4ade-8377-6ea389a3b63a-plugins-config-volume\") pod \"aab79284-1304-4ade-8377-6ea389a3b63a\" (UID: \"aab79284-1304-4ade-8377-6ea389a3b63a\") "
Oct 30 04:28:21.249768 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:21.249755    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aab79284-1304-4ade-8377-6ea389a3b63a-kube-api-access-mq9hb\" (UniqueName: \"kubernetes.io/projected/aab79284-1304-4ade-8377-6ea389a3b63a-kube-api-access-mq9hb\") pod \"aab79284-1304-4ade-8377-6ea389a3b63a\" (UID: \"aab79284-1304-4ade-8377-6ea389a3b63a\") "
Oct 30 04:28:21.249964 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:21.249945    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aab79284-1304-4ade-8377-6ea389a3b63a-plugins-config-volume" pod "aab79284-1304-4ade-8377-6ea389a3b63a" (UID: "aab79284-1304-4ade-8377-6ea389a3b63a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:28:21.250047 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:21.250031    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aab79284-1304-4ade-8377-6ea389a3b63a-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:28:21.251854 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:21.251830    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab79284-1304-4ade-8377-6ea389a3b63a-kube-api-access-mq9hb" pod "aab79284-1304-4ade-8377-6ea389a3b63a" (UID: "aab79284-1304-4ade-8377-6ea389a3b63a"). InnerVolumeSpecName "kube-api-access-mq9hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:28:21.351201 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:21.351172    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq9hb\" (UniqueName: \"kubernetes.io/projected/aab79284-1304-4ade-8377-6ea389a3b63a-kube-api-access-mq9hb\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:28:21.596898 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:21.596865    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc"]
Oct 30 04:28:21.620951 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:21.620921    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b1a417-6958676ccd-jmssc"]
Oct 30 04:28:22.736703 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:22.736669    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aab79284-1304-4ade-8377-6ea389a3b63a" path="/var/lib/kubelet/pods/aab79284-1304-4ade-8377-6ea389a3b63a/volumes"
Oct 30 04:28:23.170159 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.170089    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb"]
Oct 30 04:28:23.170652 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.170636    4874 cpu_manager.go:414] "containerMap: removing container" podUID="aab79284-1304-4ade-8377-6ea389a3b63a" containerName="payload-processing"
Oct 30 04:28:23.170701 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.170654    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aab79284-1304-4ade-8377-6ea389a3b63a" containerName="payload-processing"
Oct 30 04:28:23.170750 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.170741    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="aab79284-1304-4ade-8377-6ea389a3b63a" containerName="payload-processing"
Oct 30 04:28:23.173905 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.173890    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb"
Oct 30 04:28:23.193071 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.193047    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f7a36d\""
Oct 30 04:28:23.204445 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.204421    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f7a36d-dockercfg-g8995\""
Oct 30 04:28:23.207021 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.206998    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb"]
Oct 30 04:28:23.229229 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.229201    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9"]
Oct 30 04:28:23.232953 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.232935    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9"
Oct 30 04:28:23.250062 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.250041    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9"]
Oct 30 04:28:23.288277 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.282523    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsw54\" (UniqueName: \"kubernetes.io/projected/2feca3c5-e51b-451f-b586-38757213194b-kube-api-access-dsw54\") pod \"payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb\" (UID: \"2feca3c5-e51b-451f-b586-38757213194b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb"
Oct 30 04:28:23.288277 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.282769    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2feca3c5-e51b-451f-b586-38757213194b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb\" (UID: \"2feca3c5-e51b-451f-b586-38757213194b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb"
Oct 30 04:28:23.387383 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.386003    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f84e4d7-5178-48be-9895-48f530eacdf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9\" (UID: \"3f84e4d7-5178-48be-9895-48f530eacdf6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9"
Oct 30 04:28:23.387383 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.386080    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2feca3c5-e51b-451f-b586-38757213194b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb\" (UID: \"2feca3c5-e51b-451f-b586-38757213194b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb"
Oct 30 04:28:23.387383 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.386141    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpxw6\" (UniqueName: \"kubernetes.io/projected/3f84e4d7-5178-48be-9895-48f530eacdf6-kube-api-access-wpxw6\") pod \"payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9\" (UID: \"3f84e4d7-5178-48be-9895-48f530eacdf6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9"
Oct 30 04:28:23.387383 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.386216    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsw54\" (UniqueName: \"kubernetes.io/projected/2feca3c5-e51b-451f-b586-38757213194b-kube-api-access-dsw54\") pod \"payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb\" (UID: \"2feca3c5-e51b-451f-b586-38757213194b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb"
Oct 30 04:28:23.387688 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.387468    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2feca3c5-e51b-451f-b586-38757213194b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb\" (UID: \"2feca3c5-e51b-451f-b586-38757213194b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb"
Oct 30 04:28:23.414768 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.414733    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsw54\" (UniqueName: \"kubernetes.io/projected/2feca3c5-e51b-451f-b586-38757213194b-kube-api-access-dsw54\") pod \"payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb\" (UID: \"2feca3c5-e51b-451f-b586-38757213194b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb"
Oct 30 04:28:23.486639 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.486609    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb"
Oct 30 04:28:23.486944 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.486922    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpxw6\" (UniqueName: \"kubernetes.io/projected/3f84e4d7-5178-48be-9895-48f530eacdf6-kube-api-access-wpxw6\") pod \"payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9\" (UID: \"3f84e4d7-5178-48be-9895-48f530eacdf6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9"
Oct 30 04:28:23.487071 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.487059    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f84e4d7-5178-48be-9895-48f530eacdf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9\" (UID: \"3f84e4d7-5178-48be-9895-48f530eacdf6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9"
Oct 30 04:28:23.487827 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.487806    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f84e4d7-5178-48be-9895-48f530eacdf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9\" (UID: \"3f84e4d7-5178-48be-9895-48f530eacdf6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9"
Oct 30 04:28:23.509169 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.509146    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpxw6\" (UniqueName: \"kubernetes.io/projected/3f84e4d7-5178-48be-9895-48f530eacdf6-kube-api-access-wpxw6\") pod \"payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9\" (UID: \"3f84e4d7-5178-48be-9895-48f530eacdf6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9"
Oct 30 04:28:23.551374 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.548045    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9"
Oct 30 04:28:23.786594 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.786398    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb"]
Oct 30 04:28:23.789233 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:28:23.789208    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2feca3c5_e51b_451f_b586_38757213194b.slice/crio-76aa95ab539a5e7512252f8bf850f2b8add6a8ee68483cb53500322d44725e0b WatchSource:0}: Error finding container 76aa95ab539a5e7512252f8bf850f2b8add6a8ee68483cb53500322d44725e0b: Status 404 returned error can't find the container with id 76aa95ab539a5e7512252f8bf850f2b8add6a8ee68483cb53500322d44725e0b
Oct 30 04:28:23.868386 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:23.868359    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9"]
Oct 30 04:28:23.872290 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:28:23.872264    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f84e4d7_5178_48be_9895_48f530eacdf6.slice/crio-77e5627151c32ad15e938f23df0017ab285825de8b371de49e79d606ea62e528 WatchSource:0}: Error finding container 77e5627151c32ad15e938f23df0017ab285825de8b371de49e79d606ea62e528: Status 404 returned error can't find the container with id 77e5627151c32ad15e938f23df0017ab285825de8b371de49e79d606ea62e528
Oct 30 04:28:24.245211 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:24.245175    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9" event={"ID":"3f84e4d7-5178-48be-9895-48f530eacdf6","Type":"ContainerStarted","Data":"20fb7afe8503794218f5b960c43f007c3d54478d3e49b1b44a1e53ec14a72fcf"}
Oct 30 04:28:24.245410 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:24.245255    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9"
Oct 30 04:28:24.245410 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:24.245284    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9" event={"ID":"3f84e4d7-5178-48be-9895-48f530eacdf6","Type":"ContainerStarted","Data":"77e5627151c32ad15e938f23df0017ab285825de8b371de49e79d606ea62e528"}
Oct 30 04:28:24.246707 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:24.246680    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb" event={"ID":"2feca3c5-e51b-451f-b586-38757213194b","Type":"ContainerStarted","Data":"5dc6f642eba3e0558d05f1776048ea8e35508b28f2724a9cfdc7733be767989d"}
Oct 30 04:28:24.246813 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:24.246727    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb"
Oct 30 04:28:24.246813 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:24.246743    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb" event={"ID":"2feca3c5-e51b-451f-b586-38757213194b","Type":"ContainerStarted","Data":"76aa95ab539a5e7512252f8bf850f2b8add6a8ee68483cb53500322d44725e0b"}
Oct 30 04:28:24.285770 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:24.285726    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9" podStartSLOduration=1.2857138240000001 podStartE2EDuration="1.285713824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:28:24.283677292 +0000 UTC m=+1823.696072865" watchObservedRunningTime="2026-10-30 04:28:24.285713824 +0000 UTC m=+1823.698109397"
Oct 30 04:28:24.327173 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:24.327129    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb" podStartSLOduration=1.327118307 podStartE2EDuration="1.327118307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:28:24.32656491 +0000 UTC m=+1823.738960484" watchObservedRunningTime="2026-10-30 04:28:24.327118307 +0000 UTC m=+1823.739513871"
Oct 30 04:28:25.608846 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:28:25.608807    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:28:29.630129 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:29.630087    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v"]
Oct 30 04:28:29.642061 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:29.638518    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v"
Oct 30 04:28:29.646380 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:29.646341    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v"]
Oct 30 04:28:29.665045 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:29.665015    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ca7a81-dockercfg-h7mrf\""
Oct 30 04:28:29.665181 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:29.665079    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ca7a81\""
Oct 30 04:28:29.772067 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:29.772032    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6201c5f-2acd-4e8e-8fc6-63a7445369cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v\" (UID: \"e6201c5f-2acd-4e8e-8fc6-63a7445369cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v"
Oct 30 04:28:29.772339 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:29.772311    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfxlm\" (UniqueName: \"kubernetes.io/projected/e6201c5f-2acd-4e8e-8fc6-63a7445369cb-kube-api-access-lfxlm\") pod \"payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v\" (UID: \"e6201c5f-2acd-4e8e-8fc6-63a7445369cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v"
Oct 30 04:28:29.873848 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:29.873811    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6201c5f-2acd-4e8e-8fc6-63a7445369cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v\" (UID: \"e6201c5f-2acd-4e8e-8fc6-63a7445369cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v"
Oct 30 04:28:29.874043 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:29.873943    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfxlm\" (UniqueName: \"kubernetes.io/projected/e6201c5f-2acd-4e8e-8fc6-63a7445369cb-kube-api-access-lfxlm\") pod \"payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v\" (UID: \"e6201c5f-2acd-4e8e-8fc6-63a7445369cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v"
Oct 30 04:28:29.874635 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:29.874609    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6201c5f-2acd-4e8e-8fc6-63a7445369cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v\" (UID: \"e6201c5f-2acd-4e8e-8fc6-63a7445369cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v"
Oct 30 04:28:29.898863 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:29.898801    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfxlm\" (UniqueName: \"kubernetes.io/projected/e6201c5f-2acd-4e8e-8fc6-63a7445369cb-kube-api-access-lfxlm\") pod \"payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v\" (UID: \"e6201c5f-2acd-4e8e-8fc6-63a7445369cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v"
Oct 30 04:28:29.967378 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:29.967353    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v"
Oct 30 04:28:30.271740 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:30.271714    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v"]
Oct 30 04:28:30.274012 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:28:30.273985    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6201c5f_2acd_4e8e_8fc6_63a7445369cb.slice/crio-cee5a71f82880a33f6407b5828f6f2b34b2126de17b8cd6fd684602b1b7d5d68 WatchSource:0}: Error finding container cee5a71f82880a33f6407b5828f6f2b34b2126de17b8cd6fd684602b1b7d5d68: Status 404 returned error can't find the container with id cee5a71f82880a33f6407b5828f6f2b34b2126de17b8cd6fd684602b1b7d5d68
Oct 30 04:28:30.289821 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:30.289784    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v" event={"ID":"e6201c5f-2acd-4e8e-8fc6-63a7445369cb","Type":"ContainerStarted","Data":"cee5a71f82880a33f6407b5828f6f2b34b2126de17b8cd6fd684602b1b7d5d68"}
Oct 30 04:28:30.697337 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:30.697261    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r"]
Oct 30 04:28:30.733840 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:30.733813    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf"]
Oct 30 04:28:31.297262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:31.297210    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v" event={"ID":"e6201c5f-2acd-4e8e-8fc6-63a7445369cb","Type":"ContainerStarted","Data":"96fd018b62c0dba2da99e5f5799077fa07e0ffc89aa3731e201e0ebe88203ac0"}
Oct 30 04:28:31.297441 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:31.297398    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v"
Oct 30 04:28:31.360266 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:31.357232    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v" podStartSLOduration=2.35721071 podStartE2EDuration="2.35721071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:28:31.341507412 +0000 UTC m=+1830.753903021" watchObservedRunningTime="2026-10-30 04:28:31.35721071 +0000 UTC m=+1830.769606283"
Oct 30 04:28:35.256364 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:35.256334    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb"
Oct 30 04:28:35.256796 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:35.256497    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9"
Oct 30 04:28:35.698095 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:35.697999    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r" podUID="572594dd-6b8e-419f-a838-62671f6426c3" containerName="payload-processing" containerID="cri-o://1ef168ee1007cfee300cfb0722620268d22b636efb7ba687f5379cd622486510" gracePeriod=25
Oct 30 04:28:35.734316 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:35.734257    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf" podUID="8682db73-4941-40f5-88b6-54c04bc2212f" containerName="payload-pre-processing" containerID="cri-o://c37279ff296560eeb020c786088541ed0f94ef344990aaa71fbce7061acfa3cc" gracePeriod=25
Oct 30 04:28:35.995819 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:35.995794    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r"
Oct 30 04:28:36.042466 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.042439    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf"
Oct 30 04:28:36.054532 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.054503    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/572594dd-6b8e-419f-a838-62671f6426c3-kube-api-access-dcfsb\" (UniqueName: \"kubernetes.io/projected/572594dd-6b8e-419f-a838-62671f6426c3-kube-api-access-dcfsb\") pod \"572594dd-6b8e-419f-a838-62671f6426c3\" (UID: \"572594dd-6b8e-419f-a838-62671f6426c3\") "
Oct 30 04:28:36.054742 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.054723    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/572594dd-6b8e-419f-a838-62671f6426c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/572594dd-6b8e-419f-a838-62671f6426c3-plugins-config-volume\") pod \"572594dd-6b8e-419f-a838-62671f6426c3\" (UID: \"572594dd-6b8e-419f-a838-62671f6426c3\") "
Oct 30 04:28:36.055060 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.055038    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/572594dd-6b8e-419f-a838-62671f6426c3-plugins-config-volume" pod "572594dd-6b8e-419f-a838-62671f6426c3" (UID: "572594dd-6b8e-419f-a838-62671f6426c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:28:36.055230 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.055215    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/572594dd-6b8e-419f-a838-62671f6426c3-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:28:36.057121 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.057098    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/572594dd-6b8e-419f-a838-62671f6426c3-kube-api-access-dcfsb" pod "572594dd-6b8e-419f-a838-62671f6426c3" (UID: "572594dd-6b8e-419f-a838-62671f6426c3"). InnerVolumeSpecName "kube-api-access-dcfsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:28:36.156822 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.156788    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8682db73-4941-40f5-88b6-54c04bc2212f-kube-api-access-pzs5x\" (UniqueName: \"kubernetes.io/projected/8682db73-4941-40f5-88b6-54c04bc2212f-kube-api-access-pzs5x\") pod \"8682db73-4941-40f5-88b6-54c04bc2212f\" (UID: \"8682db73-4941-40f5-88b6-54c04bc2212f\") "
Oct 30 04:28:36.157017 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.157004    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8682db73-4941-40f5-88b6-54c04bc2212f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8682db73-4941-40f5-88b6-54c04bc2212f-plugins-config-volume\") pod \"8682db73-4941-40f5-88b6-54c04bc2212f\" (UID: \"8682db73-4941-40f5-88b6-54c04bc2212f\") "
Oct 30 04:28:36.157442 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.157413    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8682db73-4941-40f5-88b6-54c04bc2212f-plugins-config-volume" pod "8682db73-4941-40f5-88b6-54c04bc2212f" (UID: "8682db73-4941-40f5-88b6-54c04bc2212f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:28:36.157564 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.157476    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcfsb\" (UniqueName: \"kubernetes.io/projected/572594dd-6b8e-419f-a838-62671f6426c3-kube-api-access-dcfsb\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:28:36.159012 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.158981    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8682db73-4941-40f5-88b6-54c04bc2212f-kube-api-access-pzs5x" pod "8682db73-4941-40f5-88b6-54c04bc2212f" (UID: "8682db73-4941-40f5-88b6-54c04bc2212f"). InnerVolumeSpecName "kube-api-access-pzs5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:28:36.258509 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.258483    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8682db73-4941-40f5-88b6-54c04bc2212f-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:28:36.258509 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.258508    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzs5x\" (UniqueName: \"kubernetes.io/projected/8682db73-4941-40f5-88b6-54c04bc2212f-kube-api-access-pzs5x\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:28:36.328697 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.328663    4874 generic.go:356] "Generic (PLEG): container finished" podID="572594dd-6b8e-419f-a838-62671f6426c3" containerID="1ef168ee1007cfee300cfb0722620268d22b636efb7ba687f5379cd622486510" exitCode=0
Oct 30 04:28:36.328837 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.328713    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r"
Oct 30 04:28:36.328837 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.328738    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r" event={"ID":"572594dd-6b8e-419f-a838-62671f6426c3","Type":"ContainerDied","Data":"1ef168ee1007cfee300cfb0722620268d22b636efb7ba687f5379cd622486510"}
Oct 30 04:28:36.328837 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.328779    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r" event={"ID":"572594dd-6b8e-419f-a838-62671f6426c3","Type":"ContainerDied","Data":"11a735fb41e3cb660647f6112b99f796d9b97a3a6066d79a90d7365899195687"}
Oct 30 04:28:36.328837 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.328802    4874 scope.go:122] "RemoveContainer" containerID="1ef168ee1007cfee300cfb0722620268d22b636efb7ba687f5379cd622486510"
Oct 30 04:28:36.330036 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.330016    4874 generic.go:356] "Generic (PLEG): container finished" podID="8682db73-4941-40f5-88b6-54c04bc2212f" containerID="c37279ff296560eeb020c786088541ed0f94ef344990aaa71fbce7061acfa3cc" exitCode=0
Oct 30 04:28:36.330134 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.330056    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf"
Oct 30 04:28:36.330134 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.330108    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf" event={"ID":"8682db73-4941-40f5-88b6-54c04bc2212f","Type":"ContainerDied","Data":"c37279ff296560eeb020c786088541ed0f94ef344990aaa71fbce7061acfa3cc"}
Oct 30 04:28:36.330272 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.330149    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf" event={"ID":"8682db73-4941-40f5-88b6-54c04bc2212f","Type":"ContainerDied","Data":"698364a197cd5724a8d98248fb766652398ee985b9e11569b9772ab95bb36147"}
Oct 30 04:28:36.338929 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.338909    4874 scope.go:122] "RemoveContainer" containerID="1ef168ee1007cfee300cfb0722620268d22b636efb7ba687f5379cd622486510"
Oct 30 04:28:36.339276 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:28:36.339221    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ef168ee1007cfee300cfb0722620268d22b636efb7ba687f5379cd622486510\": container with ID starting with 1ef168ee1007cfee300cfb0722620268d22b636efb7ba687f5379cd622486510 not found: ID does not exist" containerID="1ef168ee1007cfee300cfb0722620268d22b636efb7ba687f5379cd622486510"
Oct 30 04:28:36.339356 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.339274    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef168ee1007cfee300cfb0722620268d22b636efb7ba687f5379cd622486510"} err="failed to get container status \"1ef168ee1007cfee300cfb0722620268d22b636efb7ba687f5379cd622486510\": rpc error: code = NotFound desc = could not find container \"1ef168ee1007cfee300cfb0722620268d22b636efb7ba687f5379cd622486510\": container with ID starting with 1ef168ee1007cfee300cfb0722620268d22b636efb7ba687f5379cd622486510 not found: ID does not exist"
Oct 30 04:28:36.339356 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.339294    4874 scope.go:122] "RemoveContainer" containerID="c37279ff296560eeb020c786088541ed0f94ef344990aaa71fbce7061acfa3cc"
Oct 30 04:28:36.347447 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.347431    4874 scope.go:122] "RemoveContainer" containerID="c37279ff296560eeb020c786088541ed0f94ef344990aaa71fbce7061acfa3cc"
Oct 30 04:28:36.347707 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:28:36.347685    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c37279ff296560eeb020c786088541ed0f94ef344990aaa71fbce7061acfa3cc\": container with ID starting with c37279ff296560eeb020c786088541ed0f94ef344990aaa71fbce7061acfa3cc not found: ID does not exist" containerID="c37279ff296560eeb020c786088541ed0f94ef344990aaa71fbce7061acfa3cc"
Oct 30 04:28:36.347770 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.347711    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37279ff296560eeb020c786088541ed0f94ef344990aaa71fbce7061acfa3cc"} err="failed to get container status \"c37279ff296560eeb020c786088541ed0f94ef344990aaa71fbce7061acfa3cc\": rpc error: code = NotFound desc = could not find container \"c37279ff296560eeb020c786088541ed0f94ef344990aaa71fbce7061acfa3cc\": container with ID starting with c37279ff296560eeb020c786088541ed0f94ef344990aaa71fbce7061acfa3cc not found: ID does not exist"
Oct 30 04:28:36.377993 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.377966    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r"]
Oct 30 04:28:36.389025 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.389003    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-763bde-77b97cc748-mw74r"]
Oct 30 04:28:36.421564 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.421542    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf"]
Oct 30 04:28:36.425194 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.425174    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-66bc8c647-qsmkf"]
Oct 30 04:28:36.729140 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.729112    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="572594dd-6b8e-419f-a838-62671f6426c3" path="/var/lib/kubelet/pods/572594dd-6b8e-419f-a838-62671f6426c3/volumes"
Oct 30 04:28:36.729517 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:36.729503    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8682db73-4941-40f5-88b6-54c04bc2212f" path="/var/lib/kubelet/pods/8682db73-4941-40f5-88b6-54c04bc2212f/volumes"
Oct 30 04:28:37.724229 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:37.722742    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw"]
Oct 30 04:28:37.724229 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:37.723525    4874 cpu_manager.go:414] "containerMap: removing container" podUID="572594dd-6b8e-419f-a838-62671f6426c3" containerName="payload-processing"
Oct 30 04:28:37.724229 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:37.723546    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="572594dd-6b8e-419f-a838-62671f6426c3" containerName="payload-processing"
Oct 30 04:28:37.724229 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:37.723575    4874 cpu_manager.go:414] "containerMap: removing container" podUID="8682db73-4941-40f5-88b6-54c04bc2212f" containerName="payload-pre-processing"
Oct 30 04:28:37.724229 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:37.723583    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8682db73-4941-40f5-88b6-54c04bc2212f" containerName="payload-pre-processing"
Oct 30 04:28:37.724229 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:37.723756    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="572594dd-6b8e-419f-a838-62671f6426c3" containerName="payload-processing"
Oct 30 04:28:37.724229 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:37.723768    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="8682db73-4941-40f5-88b6-54c04bc2212f" containerName="payload-pre-processing"
Oct 30 04:28:37.765295 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:37.765264    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw"]
Oct 30 04:28:37.767450 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:37.765652    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw"
Oct 30 04:28:37.793102 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:37.793078    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b1a417-dockercfg-gjjcw\""
Oct 30 04:28:37.793233 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:37.793098    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b1a417\""
Oct 30 04:28:37.816128 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:37.816105    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsvmx\" (UniqueName: \"kubernetes.io/projected/b03379d8-4074-4c65-b918-77ed905261a2-kube-api-access-wsvmx\") pod \"payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw\" (UID: \"b03379d8-4074-4c65-b918-77ed905261a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw"
Oct 30 04:28:37.816224 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:37.816166    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b03379d8-4074-4c65-b918-77ed905261a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw\" (UID: \"b03379d8-4074-4c65-b918-77ed905261a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw"
Oct 30 04:28:37.865994 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:37.865965    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw"]
Oct 30 04:28:37.869730 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:37.869710    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw"
Oct 30 04:28:37.889896 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:37.889870    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw"]
Oct 30 04:28:37.920373 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:37.920331    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsvmx\" (UniqueName: \"kubernetes.io/projected/b03379d8-4074-4c65-b918-77ed905261a2-kube-api-access-wsvmx\") pod \"payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw\" (UID: \"b03379d8-4074-4c65-b918-77ed905261a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw"
Oct 30 04:28:37.920521 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:37.920416    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b03379d8-4074-4c65-b918-77ed905261a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw\" (UID: \"b03379d8-4074-4c65-b918-77ed905261a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw"
Oct 30 04:28:37.921212 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:37.921183    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b03379d8-4074-4c65-b918-77ed905261a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw\" (UID: \"b03379d8-4074-4c65-b918-77ed905261a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw"
Oct 30 04:28:37.943063 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:37.943029    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsvmx\" (UniqueName: \"kubernetes.io/projected/b03379d8-4074-4c65-b918-77ed905261a2-kube-api-access-wsvmx\") pod \"payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw\" (UID: \"b03379d8-4074-4c65-b918-77ed905261a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw"
Oct 30 04:28:38.022135 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:38.022105    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqb7b\" (UniqueName: \"kubernetes.io/projected/30954ce5-9300-4778-8322-172239088b73-kube-api-access-fqb7b\") pod \"payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw\" (UID: \"30954ce5-9300-4778-8322-172239088b73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw"
Oct 30 04:28:38.022372 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:38.022350    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30954ce5-9300-4778-8322-172239088b73-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw\" (UID: \"30954ce5-9300-4778-8322-172239088b73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw"
Oct 30 04:28:38.085368 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:38.085341    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw"
Oct 30 04:28:38.123426 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:38.123391    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30954ce5-9300-4778-8322-172239088b73-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw\" (UID: \"30954ce5-9300-4778-8322-172239088b73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw"
Oct 30 04:28:38.123591 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:38.123441    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqb7b\" (UniqueName: \"kubernetes.io/projected/30954ce5-9300-4778-8322-172239088b73-kube-api-access-fqb7b\") pod \"payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw\" (UID: \"30954ce5-9300-4778-8322-172239088b73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw"
Oct 30 04:28:38.124703 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:38.124675    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30954ce5-9300-4778-8322-172239088b73-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw\" (UID: \"30954ce5-9300-4778-8322-172239088b73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw"
Oct 30 04:28:38.148536 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:38.148120    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqb7b\" (UniqueName: \"kubernetes.io/projected/30954ce5-9300-4778-8322-172239088b73-kube-api-access-fqb7b\") pod \"payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw\" (UID: \"30954ce5-9300-4778-8322-172239088b73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw"
Oct 30 04:28:38.181886 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:38.181861    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw"
Oct 30 04:28:38.385364 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:38.385340    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw"]
Oct 30 04:28:38.387941 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:28:38.387916    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03379d8_4074_4c65_b918_77ed905261a2.slice/crio-89401ddfdb31ccfe312637ff490c325fe18214f3271fdbac2bcc0e3cb3a83d77 WatchSource:0}: Error finding container 89401ddfdb31ccfe312637ff490c325fe18214f3271fdbac2bcc0e3cb3a83d77: Status 404 returned error can't find the container with id 89401ddfdb31ccfe312637ff490c325fe18214f3271fdbac2bcc0e3cb3a83d77
Oct 30 04:28:38.500265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:38.500223    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw"]
Oct 30 04:28:38.504509 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:28:38.504484    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30954ce5_9300_4778_8322_172239088b73.slice/crio-6429eba1cce4ce856e05eab4d654aef3828c48f9c2960806547078308832cd5c WatchSource:0}: Error finding container 6429eba1cce4ce856e05eab4d654aef3828c48f9c2960806547078308832cd5c: Status 404 returned error can't find the container with id 6429eba1cce4ce856e05eab4d654aef3828c48f9c2960806547078308832cd5c
Oct 30 04:28:39.349663 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:39.349629    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw" event={"ID":"30954ce5-9300-4778-8322-172239088b73","Type":"ContainerStarted","Data":"df482d307a8d850ff4f0072bf07da1042829abaf52d49e6aa63d820565e10793"}
Oct 30 04:28:39.350138 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:39.349686    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw" event={"ID":"30954ce5-9300-4778-8322-172239088b73","Type":"ContainerStarted","Data":"6429eba1cce4ce856e05eab4d654aef3828c48f9c2960806547078308832cd5c"}
Oct 30 04:28:39.350138 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:39.349715    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw"
Oct 30 04:28:39.351352 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:39.351327    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw" event={"ID":"b03379d8-4074-4c65-b918-77ed905261a2","Type":"ContainerStarted","Data":"67c554962b7daf8a509436bab474381931c1874ede20b7f528ef5bdc1d431a45"}
Oct 30 04:28:39.351462 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:39.351378    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw" event={"ID":"b03379d8-4074-4c65-b918-77ed905261a2","Type":"ContainerStarted","Data":"89401ddfdb31ccfe312637ff490c325fe18214f3271fdbac2bcc0e3cb3a83d77"}
Oct 30 04:28:39.351462 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:39.351399    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw"
Oct 30 04:28:39.391457 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:39.391413    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw" podStartSLOduration=2.391400018 podStartE2EDuration="2.391400018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:28:39.388895509 +0000 UTC m=+1838.801291081" watchObservedRunningTime="2026-10-30 04:28:39.391400018 +0000 UTC m=+1838.803795640"
Oct 30 04:28:39.429755 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:39.429701    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw" podStartSLOduration=2.429684768 podStartE2EDuration="2.429684768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:28:39.42770247 +0000 UTC m=+1838.840098044" watchObservedRunningTime="2026-10-30 04:28:39.429684768 +0000 UTC m=+1838.842080342"
Oct 30 04:28:42.305068 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:42.305036    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v"
Oct 30 04:28:50.358774 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:50.358743    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw"
Oct 30 04:28:50.359204 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:50.358813    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw"
Oct 30 04:28:51.360621 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:51.360582    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz"]
Oct 30 04:28:51.368000 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:51.367966    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz"
Oct 30 04:28:51.378420 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:51.377803    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz"]
Oct 30 04:28:51.385752 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:51.385723    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-763bde\""
Oct 30 04:28:51.386313 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:51.386290    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-763bde-dockercfg-s87z4\""
Oct 30 04:28:51.397054 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:51.397022    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzbq6\" (UniqueName: \"kubernetes.io/projected/54704513-6246-41c9-978f-77b8d99e085b-kube-api-access-vzbq6\") pod \"payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz\" (UID: \"54704513-6246-41c9-978f-77b8d99e085b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz"
Oct 30 04:28:51.397251 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:51.397221    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54704513-6246-41c9-978f-77b8d99e085b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz\" (UID: \"54704513-6246-41c9-978f-77b8d99e085b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz"
Oct 30 04:28:51.498216 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:51.498170    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54704513-6246-41c9-978f-77b8d99e085b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz\" (UID: \"54704513-6246-41c9-978f-77b8d99e085b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz"
Oct 30 04:28:51.498416 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:51.498357    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzbq6\" (UniqueName: \"kubernetes.io/projected/54704513-6246-41c9-978f-77b8d99e085b-kube-api-access-vzbq6\") pod \"payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz\" (UID: \"54704513-6246-41c9-978f-77b8d99e085b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz"
Oct 30 04:28:51.498855 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:51.498835    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54704513-6246-41c9-978f-77b8d99e085b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz\" (UID: \"54704513-6246-41c9-978f-77b8d99e085b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz"
Oct 30 04:28:51.518826 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:51.518798    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk"]
Oct 30 04:28:51.522876 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:51.522860    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk"
Oct 30 04:28:51.525792 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:51.525770    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzbq6\" (UniqueName: \"kubernetes.io/projected/54704513-6246-41c9-978f-77b8d99e085b-kube-api-access-vzbq6\") pod \"payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz\" (UID: \"54704513-6246-41c9-978f-77b8d99e085b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz"
Oct 30 04:28:51.544356 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:51.544333    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk"]
Oct 30 04:28:51.599694 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:51.599660    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t4sd\" (UniqueName: \"kubernetes.io/projected/d128d0ee-3fa4-4023-9333-cb7fa4ff5940-kube-api-access-7t4sd\") pod \"payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk\" (UID: \"d128d0ee-3fa4-4023-9333-cb7fa4ff5940\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk"
Oct 30 04:28:51.599873 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:51.599727    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d128d0ee-3fa4-4023-9333-cb7fa4ff5940-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk\" (UID: \"d128d0ee-3fa4-4023-9333-cb7fa4ff5940\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk"
Oct 30 04:28:51.693076 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:51.692995    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz"
Oct 30 04:28:51.703514 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:51.703485    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t4sd\" (UniqueName: \"kubernetes.io/projected/d128d0ee-3fa4-4023-9333-cb7fa4ff5940-kube-api-access-7t4sd\") pod \"payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk\" (UID: \"d128d0ee-3fa4-4023-9333-cb7fa4ff5940\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk"
Oct 30 04:28:51.703665 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:51.703651    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d128d0ee-3fa4-4023-9333-cb7fa4ff5940-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk\" (UID: \"d128d0ee-3fa4-4023-9333-cb7fa4ff5940\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk"
Oct 30 04:28:51.704376 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:51.704349    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d128d0ee-3fa4-4023-9333-cb7fa4ff5940-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk\" (UID: \"d128d0ee-3fa4-4023-9333-cb7fa4ff5940\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk"
Oct 30 04:28:51.726329 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:51.726293    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t4sd\" (UniqueName: \"kubernetes.io/projected/d128d0ee-3fa4-4023-9333-cb7fa4ff5940-kube-api-access-7t4sd\") pod \"payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk\" (UID: \"d128d0ee-3fa4-4023-9333-cb7fa4ff5940\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk"
Oct 30 04:28:51.847061 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:51.847028    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk"
Oct 30 04:28:52.001900 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:52.001878    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz"]
Oct 30 04:28:52.004254 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:28:52.004211    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54704513_6246_41c9_978f_77b8d99e085b.slice/crio-cc2cacb78a10b0c79308aed9648a0514de8de8cb070d78b69abad2e1a46703f7 WatchSource:0}: Error finding container cc2cacb78a10b0c79308aed9648a0514de8de8cb070d78b69abad2e1a46703f7: Status 404 returned error can't find the container with id cc2cacb78a10b0c79308aed9648a0514de8de8cb070d78b69abad2e1a46703f7
Oct 30 04:28:52.158233 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:52.158205    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk"]
Oct 30 04:28:52.162573 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:28:52.162546    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd128d0ee_3fa4_4023_9333_cb7fa4ff5940.slice/crio-cd82df6b74e64a7b4270a5e16e745cf515e1422dad6e1202fab6768fed4b6cab WatchSource:0}: Error finding container cd82df6b74e64a7b4270a5e16e745cf515e1422dad6e1202fab6768fed4b6cab: Status 404 returned error can't find the container with id cd82df6b74e64a7b4270a5e16e745cf515e1422dad6e1202fab6768fed4b6cab
Oct 30 04:28:52.432815 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:52.432721    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk" event={"ID":"d128d0ee-3fa4-4023-9333-cb7fa4ff5940","Type":"ContainerStarted","Data":"16cc713a89d924e1831ee15a27825a562ada950d42b8ded553dc5bc50e141c94"}
Oct 30 04:28:52.432815 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:52.432783    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk" event={"ID":"d128d0ee-3fa4-4023-9333-cb7fa4ff5940","Type":"ContainerStarted","Data":"cd82df6b74e64a7b4270a5e16e745cf515e1422dad6e1202fab6768fed4b6cab"}
Oct 30 04:28:52.433327 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:52.432913    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk"
Oct 30 04:28:52.434275 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:52.434226    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz" event={"ID":"54704513-6246-41c9-978f-77b8d99e085b","Type":"ContainerStarted","Data":"de1a7fed98e7620a237d9b0f21072f7c2fa3f76afbdb4366a3a37c1e81449a19"}
Oct 30 04:28:52.434408 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:52.434305    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz"
Oct 30 04:28:52.434408 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:52.434329    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz" event={"ID":"54704513-6246-41c9-978f-77b8d99e085b","Type":"ContainerStarted","Data":"cc2cacb78a10b0c79308aed9648a0514de8de8cb070d78b69abad2e1a46703f7"}
Oct 30 04:28:52.473026 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:52.472981    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk" podStartSLOduration=1.472969318 podStartE2EDuration="1.472969318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:28:52.471503152 +0000 UTC m=+1851.883898725" watchObservedRunningTime="2026-10-30 04:28:52.472969318 +0000 UTC m=+1851.885364890"
Oct 30 04:28:52.511466 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:52.511426    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz" podStartSLOduration=1.511413814 podStartE2EDuration="1.511413814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:28:52.510960792 +0000 UTC m=+1851.923356365" watchObservedRunningTime="2026-10-30 04:28:52.511413814 +0000 UTC m=+1851.923809392"
Oct 30 04:28:52.674270 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:28:52.674222    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:28:55.609374 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:28:55.609336    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:29:03.448003 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:03.447973    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk"
Oct 30 04:29:03.448418 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:03.448035    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz"
Oct 30 04:29:13.955989 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:13.955956    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7"]
Oct 30 04:29:13.956510 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:13.956257    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7" podUID="ed9a9630-d7e9-4480-8139-0fcd83c56c32" containerName="maas-api" containerID="cri-o://e2d1257a7cc1f67d6b7a3a3040b523543b5a106fb3fc5e230aa88dcc80e57a3a" gracePeriod=30
Oct 30 04:29:14.231745 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:14.231722    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7"
Oct 30 04:29:14.249623 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:14.249598    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed9a9630-d7e9-4480-8139-0fcd83c56c32-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9a9630-d7e9-4480-8139-0fcd83c56c32-openshift-service-ca\") pod \"ed9a9630-d7e9-4480-8139-0fcd83c56c32\" (UID: \"ed9a9630-d7e9-4480-8139-0fcd83c56c32\") "
Oct 30 04:29:14.249728 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:14.249708    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed9a9630-d7e9-4480-8139-0fcd83c56c32-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed9a9630-d7e9-4480-8139-0fcd83c56c32-maas-api-tls\") pod \"ed9a9630-d7e9-4480-8139-0fcd83c56c32\" (UID: \"ed9a9630-d7e9-4480-8139-0fcd83c56c32\") "
Oct 30 04:29:14.249848 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:14.249832    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed9a9630-d7e9-4480-8139-0fcd83c56c32-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed9a9630-d7e9-4480-8139-0fcd83c56c32-metrics-certs\") pod \"ed9a9630-d7e9-4480-8139-0fcd83c56c32\" (UID: \"ed9a9630-d7e9-4480-8139-0fcd83c56c32\") "
Oct 30 04:29:14.249902 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:14.249879    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed9a9630-d7e9-4480-8139-0fcd83c56c32-kube-api-access-rxf86\" (UniqueName: \"kubernetes.io/projected/ed9a9630-d7e9-4480-8139-0fcd83c56c32-kube-api-access-rxf86\") pod \"ed9a9630-d7e9-4480-8139-0fcd83c56c32\" (UID: \"ed9a9630-d7e9-4480-8139-0fcd83c56c32\") "
Oct 30 04:29:14.250040 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:14.249901    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed9a9630-d7e9-4480-8139-0fcd83c56c32-openshift-service-ca" pod "ed9a9630-d7e9-4480-8139-0fcd83c56c32" (UID: "ed9a9630-d7e9-4480-8139-0fcd83c56c32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:29:14.250427 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:14.250405    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9a9630-d7e9-4480-8139-0fcd83c56c32-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:14.251913 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:14.251861    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed9a9630-d7e9-4480-8139-0fcd83c56c32-maas-api-tls" pod "ed9a9630-d7e9-4480-8139-0fcd83c56c32" (UID: "ed9a9630-d7e9-4480-8139-0fcd83c56c32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:29:14.251913 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:14.251902    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed9a9630-d7e9-4480-8139-0fcd83c56c32-metrics-certs" pod "ed9a9630-d7e9-4480-8139-0fcd83c56c32" (UID: "ed9a9630-d7e9-4480-8139-0fcd83c56c32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:29:14.252031 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:14.251992    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9a9630-d7e9-4480-8139-0fcd83c56c32-kube-api-access-rxf86" pod "ed9a9630-d7e9-4480-8139-0fcd83c56c32" (UID: "ed9a9630-d7e9-4480-8139-0fcd83c56c32"). InnerVolumeSpecName "kube-api-access-rxf86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:29:14.351221 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:14.351187    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed9a9630-d7e9-4480-8139-0fcd83c56c32-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:14.351221 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:14.351216    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxf86\" (UniqueName: \"kubernetes.io/projected/ed9a9630-d7e9-4480-8139-0fcd83c56c32-kube-api-access-rxf86\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:14.351405 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:14.351229    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed9a9630-d7e9-4480-8139-0fcd83c56c32-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:14.576389 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:14.576354    4874 generic.go:356] "Generic (PLEG): container finished" podID="ed9a9630-d7e9-4480-8139-0fcd83c56c32" containerID="e2d1257a7cc1f67d6b7a3a3040b523543b5a106fb3fc5e230aa88dcc80e57a3a" exitCode=0
Oct 30 04:29:14.576633 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:14.576413    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7"
Oct 30 04:29:14.576633 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:14.576448    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7" event={"ID":"ed9a9630-d7e9-4480-8139-0fcd83c56c32","Type":"ContainerDied","Data":"e2d1257a7cc1f67d6b7a3a3040b523543b5a106fb3fc5e230aa88dcc80e57a3a"}
Oct 30 04:29:14.576633 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:14.576517    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7" event={"ID":"ed9a9630-d7e9-4480-8139-0fcd83c56c32","Type":"ContainerDied","Data":"d3a860a831276e8fc563217d6856a19cf2c9311e5d450178265c28f8cac5427c"}
Oct 30 04:29:14.576633 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:14.576555    4874 scope.go:122] "RemoveContainer" containerID="e2d1257a7cc1f67d6b7a3a3040b523543b5a106fb3fc5e230aa88dcc80e57a3a"
Oct 30 04:29:14.591449 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:14.591405    4874 scope.go:122] "RemoveContainer" containerID="e2d1257a7cc1f67d6b7a3a3040b523543b5a106fb3fc5e230aa88dcc80e57a3a"
Oct 30 04:29:14.591719 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:29:14.591694    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2d1257a7cc1f67d6b7a3a3040b523543b5a106fb3fc5e230aa88dcc80e57a3a\": container with ID starting with e2d1257a7cc1f67d6b7a3a3040b523543b5a106fb3fc5e230aa88dcc80e57a3a not found: ID does not exist" containerID="e2d1257a7cc1f67d6b7a3a3040b523543b5a106fb3fc5e230aa88dcc80e57a3a"
Oct 30 04:29:14.591809 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:14.591724    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2d1257a7cc1f67d6b7a3a3040b523543b5a106fb3fc5e230aa88dcc80e57a3a"} err="failed to get container status \"e2d1257a7cc1f67d6b7a3a3040b523543b5a106fb3fc5e230aa88dcc80e57a3a\": rpc error: code = NotFound desc = could not find container \"e2d1257a7cc1f67d6b7a3a3040b523543b5a106fb3fc5e230aa88dcc80e57a3a\": container with ID starting with e2d1257a7cc1f67d6b7a3a3040b523543b5a106fb3fc5e230aa88dcc80e57a3a not found: ID does not exist"
Oct 30 04:29:14.626096 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:14.626073    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7"]
Oct 30 04:29:14.632655 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:14.632634    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf4238-75785cff95-hz2t7"]
Oct 30 04:29:14.729722 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:14.729697    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed9a9630-d7e9-4480-8139-0fcd83c56c32" path="/var/lib/kubelet/pods/ed9a9630-d7e9-4480-8139-0fcd83c56c32/volumes"
Oct 30 04:29:17.365701 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:17.365664    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb"]
Oct 30 04:29:17.397524 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:17.397495    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9"]
Oct 30 04:29:21.804261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:21.796135    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:29:22.366410 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:22.366369    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb" podUID="2feca3c5-e51b-451f-b586-38757213194b" containerName="payload-processing" containerID="cri-o://5dc6f642eba3e0558d05f1776048ea8e35508b28f2724a9cfdc7733be767989d" gracePeriod=25
Oct 30 04:29:22.398050 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:22.398017    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9" podUID="3f84e4d7-5178-48be-9895-48f530eacdf6" containerName="payload-pre-processing" containerID="cri-o://20fb7afe8503794218f5b960c43f007c3d54478d3e49b1b44a1e53ec14a72fcf" gracePeriod=25
Oct 30 04:29:22.673037 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:22.672991    4874 generic.go:356] "Generic (PLEG): container finished" podID="2feca3c5-e51b-451f-b586-38757213194b" containerID="5dc6f642eba3e0558d05f1776048ea8e35508b28f2724a9cfdc7733be767989d" exitCode=0
Oct 30 04:29:22.673278 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:22.673123    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb" event={"ID":"2feca3c5-e51b-451f-b586-38757213194b","Type":"ContainerDied","Data":"5dc6f642eba3e0558d05f1776048ea8e35508b28f2724a9cfdc7733be767989d"}
Oct 30 04:29:22.673278 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:22.673173    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb" event={"ID":"2feca3c5-e51b-451f-b586-38757213194b","Type":"ContainerDied","Data":"76aa95ab539a5e7512252f8bf850f2b8add6a8ee68483cb53500322d44725e0b"}
Oct 30 04:29:22.673278 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:22.673199    4874 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76aa95ab539a5e7512252f8bf850f2b8add6a8ee68483cb53500322d44725e0b"
Oct 30 04:29:22.675106 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:22.675083    4874 generic.go:356] "Generic (PLEG): container finished" podID="3f84e4d7-5178-48be-9895-48f530eacdf6" containerID="20fb7afe8503794218f5b960c43f007c3d54478d3e49b1b44a1e53ec14a72fcf" exitCode=0
Oct 30 04:29:22.675230 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:22.675156    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9" event={"ID":"3f84e4d7-5178-48be-9895-48f530eacdf6","Type":"ContainerDied","Data":"20fb7afe8503794218f5b960c43f007c3d54478d3e49b1b44a1e53ec14a72fcf"}
Oct 30 04:29:22.685275 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:22.685256    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb"
Oct 30 04:29:22.704612 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:22.704591    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9"
Oct 30 04:29:22.813803 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:22.813766    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2feca3c5-e51b-451f-b586-38757213194b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2feca3c5-e51b-451f-b586-38757213194b-plugins-config-volume\") pod \"2feca3c5-e51b-451f-b586-38757213194b\" (UID: \"2feca3c5-e51b-451f-b586-38757213194b\") "
Oct 30 04:29:22.813803 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:22.813812    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f84e4d7-5178-48be-9895-48f530eacdf6-kube-api-access-wpxw6\" (UniqueName: \"kubernetes.io/projected/3f84e4d7-5178-48be-9895-48f530eacdf6-kube-api-access-wpxw6\") pod \"3f84e4d7-5178-48be-9895-48f530eacdf6\" (UID: \"3f84e4d7-5178-48be-9895-48f530eacdf6\") "
Oct 30 04:29:22.814357 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:22.813838    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2feca3c5-e51b-451f-b586-38757213194b-kube-api-access-dsw54\" (UniqueName: \"kubernetes.io/projected/2feca3c5-e51b-451f-b586-38757213194b-kube-api-access-dsw54\") pod \"2feca3c5-e51b-451f-b586-38757213194b\" (UID: \"2feca3c5-e51b-451f-b586-38757213194b\") "
Oct 30 04:29:22.814357 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:22.813936    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f84e4d7-5178-48be-9895-48f530eacdf6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f84e4d7-5178-48be-9895-48f530eacdf6-plugins-config-volume\") pod \"3f84e4d7-5178-48be-9895-48f530eacdf6\" (UID: \"3f84e4d7-5178-48be-9895-48f530eacdf6\") "
Oct 30 04:29:22.814357 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:22.814277    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2feca3c5-e51b-451f-b586-38757213194b-plugins-config-volume" pod "2feca3c5-e51b-451f-b586-38757213194b" (UID: "2feca3c5-e51b-451f-b586-38757213194b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:29:22.814357 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:22.814331    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f84e4d7-5178-48be-9895-48f530eacdf6-plugins-config-volume" pod "3f84e4d7-5178-48be-9895-48f530eacdf6" (UID: "3f84e4d7-5178-48be-9895-48f530eacdf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:29:22.814661 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:22.814639    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2feca3c5-e51b-451f-b586-38757213194b-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:22.814751 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:22.814668    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f84e4d7-5178-48be-9895-48f530eacdf6-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:22.816090 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:22.816068    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2feca3c5-e51b-451f-b586-38757213194b-kube-api-access-dsw54" pod "2feca3c5-e51b-451f-b586-38757213194b" (UID: "2feca3c5-e51b-451f-b586-38757213194b"). InnerVolumeSpecName "kube-api-access-dsw54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:29:22.816090 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:22.816077    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f84e4d7-5178-48be-9895-48f530eacdf6-kube-api-access-wpxw6" pod "3f84e4d7-5178-48be-9895-48f530eacdf6" (UID: "3f84e4d7-5178-48be-9895-48f530eacdf6"). InnerVolumeSpecName "kube-api-access-wpxw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:29:22.915865 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:22.915799    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpxw6\" (UniqueName: \"kubernetes.io/projected/3f84e4d7-5178-48be-9895-48f530eacdf6-kube-api-access-wpxw6\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:22.915865 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:22.915823    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsw54\" (UniqueName: \"kubernetes.io/projected/2feca3c5-e51b-451f-b586-38757213194b-kube-api-access-dsw54\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:23.681738 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:23.681661    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9"
Oct 30 04:29:23.681738 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:23.681678    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9" event={"ID":"3f84e4d7-5178-48be-9895-48f530eacdf6","Type":"ContainerDied","Data":"77e5627151c32ad15e938f23df0017ab285825de8b371de49e79d606ea62e528"}
Oct 30 04:29:23.681738 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:23.681737    4874 scope.go:122] "RemoveContainer" containerID="20fb7afe8503794218f5b960c43f007c3d54478d3e49b1b44a1e53ec14a72fcf"
Oct 30 04:29:23.681983 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:23.681672    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb"
Oct 30 04:29:23.765032 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:23.765001    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9"]
Oct 30 04:29:23.775453 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:23.775429    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-75986dd747-f7rf9"]
Oct 30 04:29:23.799864 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:23.799834    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb"]
Oct 30 04:29:23.803968 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:23.803947    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-68dc5cdcbb-b5tbb"]
Oct 30 04:29:24.729416 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:24.729384    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2feca3c5-e51b-451f-b586-38757213194b" path="/var/lib/kubelet/pods/2feca3c5-e51b-451f-b586-38757213194b/volumes"
Oct 30 04:29:24.729785 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:24.729763    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f84e4d7-5178-48be-9895-48f530eacdf6" path="/var/lib/kubelet/pods/3f84e4d7-5178-48be-9895-48f530eacdf6/volumes"
Oct 30 04:29:24.980382 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:24.980306    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q"]
Oct 30 04:29:24.980829 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:24.980817    4874 cpu_manager.go:414] "containerMap: removing container" podUID="ed9a9630-d7e9-4480-8139-0fcd83c56c32" containerName="maas-api"
Oct 30 04:29:24.980870 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:24.980830    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed9a9630-d7e9-4480-8139-0fcd83c56c32" containerName="maas-api"
Oct 30 04:29:24.980870 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:24.980841    4874 cpu_manager.go:414] "containerMap: removing container" podUID="2feca3c5-e51b-451f-b586-38757213194b" containerName="payload-processing"
Oct 30 04:29:24.980870 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:24.980846    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2feca3c5-e51b-451f-b586-38757213194b" containerName="payload-processing"
Oct 30 04:29:24.980966 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:24.980870    4874 cpu_manager.go:414] "containerMap: removing container" podUID="3f84e4d7-5178-48be-9895-48f530eacdf6" containerName="payload-pre-processing"
Oct 30 04:29:24.980966 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:24.980875    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f84e4d7-5178-48be-9895-48f530eacdf6" containerName="payload-pre-processing"
Oct 30 04:29:24.980966 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:24.980960    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f84e4d7-5178-48be-9895-48f530eacdf6" containerName="payload-pre-processing"
Oct 30 04:29:24.981057 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:24.980970    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed9a9630-d7e9-4480-8139-0fcd83c56c32" containerName="maas-api"
Oct 30 04:29:24.981057 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:24.980977    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="2feca3c5-e51b-451f-b586-38757213194b" containerName="payload-processing"
Oct 30 04:29:24.984501 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:24.984486    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q"
Oct 30 04:29:24.998703 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:24.998685    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3f5508-serving-cert\""
Oct 30 04:29:25.022718 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:25.022688    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q"]
Oct 30 04:29:25.149443 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:25.149415    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n4mb\" (UniqueName: \"kubernetes.io/projected/8ee1f84a-09e0-483f-aea8-10ad797b3201-kube-api-access-4n4mb\") pod \"maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q\" (UID: \"8ee1f84a-09e0-483f-aea8-10ad797b3201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q"
Oct 30 04:29:25.149606 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:25.149576    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ee1f84a-09e0-483f-aea8-10ad797b3201-metrics-certs\") pod \"maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q\" (UID: \"8ee1f84a-09e0-483f-aea8-10ad797b3201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q"
Oct 30 04:29:25.149670 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:25.149606    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee1f84a-09e0-483f-aea8-10ad797b3201-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q\" (UID: \"8ee1f84a-09e0-483f-aea8-10ad797b3201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q"
Oct 30 04:29:25.149782 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:25.149758    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ee1f84a-09e0-483f-aea8-10ad797b3201-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q\" (UID: \"8ee1f84a-09e0-483f-aea8-10ad797b3201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q"
Oct 30 04:29:25.251384 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:25.251288    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n4mb\" (UniqueName: \"kubernetes.io/projected/8ee1f84a-09e0-483f-aea8-10ad797b3201-kube-api-access-4n4mb\") pod \"maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q\" (UID: \"8ee1f84a-09e0-483f-aea8-10ad797b3201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q"
Oct 30 04:29:25.251561 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:25.251479    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ee1f84a-09e0-483f-aea8-10ad797b3201-metrics-certs\") pod \"maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q\" (UID: \"8ee1f84a-09e0-483f-aea8-10ad797b3201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q"
Oct 30 04:29:25.251561 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:25.251522    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee1f84a-09e0-483f-aea8-10ad797b3201-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q\" (UID: \"8ee1f84a-09e0-483f-aea8-10ad797b3201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q"
Oct 30 04:29:25.251706 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:25.251628    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ee1f84a-09e0-483f-aea8-10ad797b3201-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q\" (UID: \"8ee1f84a-09e0-483f-aea8-10ad797b3201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q"
Oct 30 04:29:25.252219 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:25.252192    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee1f84a-09e0-483f-aea8-10ad797b3201-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q\" (UID: \"8ee1f84a-09e0-483f-aea8-10ad797b3201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q"
Oct 30 04:29:25.253987 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:25.253962    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ee1f84a-09e0-483f-aea8-10ad797b3201-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q\" (UID: \"8ee1f84a-09e0-483f-aea8-10ad797b3201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q"
Oct 30 04:29:25.254086 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:25.253993    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ee1f84a-09e0-483f-aea8-10ad797b3201-metrics-certs\") pod \"maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q\" (UID: \"8ee1f84a-09e0-483f-aea8-10ad797b3201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q"
Oct 30 04:29:25.273497 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:25.273474    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n4mb\" (UniqueName: \"kubernetes.io/projected/8ee1f84a-09e0-483f-aea8-10ad797b3201-kube-api-access-4n4mb\") pod \"maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q\" (UID: \"8ee1f84a-09e0-483f-aea8-10ad797b3201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q"
Oct 30 04:29:25.296789 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:25.296770    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q"
Oct 30 04:29:25.600133 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:25.600105    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q"]
Oct 30 04:29:25.601746 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:29:25.601716    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee1f84a_09e0_483f_aea8_10ad797b3201.slice/crio-b072da8530b33fbaed3e6b8078108029f4b6dd56c2a76f29accacf088b405d30 WatchSource:0}: Error finding container b072da8530b33fbaed3e6b8078108029f4b6dd56c2a76f29accacf088b405d30: Status 404 returned error can't find the container with id b072da8530b33fbaed3e6b8078108029f4b6dd56c2a76f29accacf088b405d30
Oct 30 04:29:25.603182 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:25.603151    4874 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 04:29:25.612750 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:29:25.612722    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:29:25.694417 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:25.694389    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q" event={"ID":"8ee1f84a-09e0-483f-aea8-10ad797b3201","Type":"ContainerStarted","Data":"b072da8530b33fbaed3e6b8078108029f4b6dd56c2a76f29accacf088b405d30"}
Oct 30 04:29:26.700461 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:26.700429    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg"]
Oct 30 04:29:26.700955 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:26.700677    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg" podUID="9c67bdeb-196a-4314-a541-48cbac967a46" containerName="maas-api" containerID="cri-o://97bafc537f8289c7246aa3dd8b35d90bc2cb5e76e7e8c9ff6a2d76e746a7a021" gracePeriod=30
Oct 30 04:29:26.702593 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:26.702556    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q" event={"ID":"8ee1f84a-09e0-483f-aea8-10ad797b3201","Type":"ContainerStarted","Data":"cfde6e72ff02db4e42f6abca02663229d54987f4b481acd477f80124d6af2b59"}
Oct 30 04:29:26.703295 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:26.703272    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q"
Oct 30 04:29:26.788251 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:26.788195    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q" podStartSLOduration=2.422290606 podStartE2EDuration="2.788180449s" totalImagesPullingTime="365.889843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:29:26.78571726 +0000 UTC m=+1886.198112834" watchObservedRunningTime="2026-10-30 04:29:26.788180449 +0000 UTC m=+1886.200576036"
Oct 30 04:29:26.982626 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:26.982593    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg"
Oct 30 04:29:27.077819 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:27.077785    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c67bdeb-196a-4314-a541-48cbac967a46-kube-api-access-swxcg\" (UniqueName: \"kubernetes.io/projected/9c67bdeb-196a-4314-a541-48cbac967a46-kube-api-access-swxcg\") pod \"9c67bdeb-196a-4314-a541-48cbac967a46\" (UID: \"9c67bdeb-196a-4314-a541-48cbac967a46\") "
Oct 30 04:29:27.077819 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:27.077824    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c67bdeb-196a-4314-a541-48cbac967a46-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c67bdeb-196a-4314-a541-48cbac967a46-maas-api-tls\") pod \"9c67bdeb-196a-4314-a541-48cbac967a46\" (UID: \"9c67bdeb-196a-4314-a541-48cbac967a46\") "
Oct 30 04:29:27.078063 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:27.077877    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c67bdeb-196a-4314-a541-48cbac967a46-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c67bdeb-196a-4314-a541-48cbac967a46-metrics-certs\") pod \"9c67bdeb-196a-4314-a541-48cbac967a46\" (UID: \"9c67bdeb-196a-4314-a541-48cbac967a46\") "
Oct 30 04:29:27.078063 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:27.077911    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c67bdeb-196a-4314-a541-48cbac967a46-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c67bdeb-196a-4314-a541-48cbac967a46-openshift-service-ca\") pod \"9c67bdeb-196a-4314-a541-48cbac967a46\" (UID: \"9c67bdeb-196a-4314-a541-48cbac967a46\") "
Oct 30 04:29:27.078333 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:27.078307    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c67bdeb-196a-4314-a541-48cbac967a46-openshift-service-ca" pod "9c67bdeb-196a-4314-a541-48cbac967a46" (UID: "9c67bdeb-196a-4314-a541-48cbac967a46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:29:27.080062 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:27.080032    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c67bdeb-196a-4314-a541-48cbac967a46-metrics-certs" pod "9c67bdeb-196a-4314-a541-48cbac967a46" (UID: "9c67bdeb-196a-4314-a541-48cbac967a46"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:29:27.080159 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:27.080057    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c67bdeb-196a-4314-a541-48cbac967a46-maas-api-tls" pod "9c67bdeb-196a-4314-a541-48cbac967a46" (UID: "9c67bdeb-196a-4314-a541-48cbac967a46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:29:27.080159 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:27.080078    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c67bdeb-196a-4314-a541-48cbac967a46-kube-api-access-swxcg" pod "9c67bdeb-196a-4314-a541-48cbac967a46" (UID: "9c67bdeb-196a-4314-a541-48cbac967a46"). InnerVolumeSpecName "kube-api-access-swxcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:29:27.178848 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:27.178821    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swxcg\" (UniqueName: \"kubernetes.io/projected/9c67bdeb-196a-4314-a541-48cbac967a46-kube-api-access-swxcg\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:27.178848 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:27.178848    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c67bdeb-196a-4314-a541-48cbac967a46-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:27.179010 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:27.178858    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c67bdeb-196a-4314-a541-48cbac967a46-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:27.179010 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:27.178866    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c67bdeb-196a-4314-a541-48cbac967a46-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:27.758138 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:27.758104    4874 generic.go:356] "Generic (PLEG): container finished" podID="9c67bdeb-196a-4314-a541-48cbac967a46" containerID="97bafc537f8289c7246aa3dd8b35d90bc2cb5e76e7e8c9ff6a2d76e746a7a021" exitCode=0
Oct 30 04:29:27.758792 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:27.758140    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg" event={"ID":"9c67bdeb-196a-4314-a541-48cbac967a46","Type":"ContainerDied","Data":"97bafc537f8289c7246aa3dd8b35d90bc2cb5e76e7e8c9ff6a2d76e746a7a021"}
Oct 30 04:29:27.758873 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:27.758833    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg" event={"ID":"9c67bdeb-196a-4314-a541-48cbac967a46","Type":"ContainerDied","Data":"79af2f81be04dac4b3ea485cdefeeda36f4ad74dd14e239d19cf246c7589cf9e"}
Oct 30 04:29:27.758873 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:27.758867    4874 scope.go:122] "RemoveContainer" containerID="97bafc537f8289c7246aa3dd8b35d90bc2cb5e76e7e8c9ff6a2d76e746a7a021"
Oct 30 04:29:27.758983 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:27.758220    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg"
Oct 30 04:29:27.770888 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:27.770872    4874 scope.go:122] "RemoveContainer" containerID="97bafc537f8289c7246aa3dd8b35d90bc2cb5e76e7e8c9ff6a2d76e746a7a021"
Oct 30 04:29:27.771161 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:29:27.771138    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97bafc537f8289c7246aa3dd8b35d90bc2cb5e76e7e8c9ff6a2d76e746a7a021\": container with ID starting with 97bafc537f8289c7246aa3dd8b35d90bc2cb5e76e7e8c9ff6a2d76e746a7a021 not found: ID does not exist" containerID="97bafc537f8289c7246aa3dd8b35d90bc2cb5e76e7e8c9ff6a2d76e746a7a021"
Oct 30 04:29:27.771290 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:27.771168    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97bafc537f8289c7246aa3dd8b35d90bc2cb5e76e7e8c9ff6a2d76e746a7a021"} err="failed to get container status \"97bafc537f8289c7246aa3dd8b35d90bc2cb5e76e7e8c9ff6a2d76e746a7a021\": rpc error: code = NotFound desc = could not find container \"97bafc537f8289c7246aa3dd8b35d90bc2cb5e76e7e8c9ff6a2d76e746a7a021\": container with ID starting with 97bafc537f8289c7246aa3dd8b35d90bc2cb5e76e7e8c9ff6a2d76e746a7a021 not found: ID does not exist"
Oct 30 04:29:27.808880 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:27.808853    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg"]
Oct 30 04:29:27.821130 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:27.821107    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6dcea5-6958894ccd-x2wxg"]
Oct 30 04:29:28.731582 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:28.731550    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c67bdeb-196a-4314-a541-48cbac967a46" path="/var/lib/kubelet/pods/9c67bdeb-196a-4314-a541-48cbac967a46/volumes"
Oct 30 04:29:33.256970 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:33.256936    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f"]
Oct 30 04:29:33.257425 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:33.257112    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f" podUID="83b9efdd-4a34-428e-9f19-3ad6be585fe0" containerName="payload-processing" containerID="cri-o://55aa0b0bd26227f078b51eaad0ce87dedc9bb0ef982bf092e3b4947f146ddeae" gracePeriod=30
Oct 30 04:29:33.530540 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:33.530512    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f"
Oct 30 04:29:33.676262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:33.658767    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b9efdd-4a34-428e-9f19-3ad6be585fe0-kube-api-access-xm99z\" (UniqueName: \"kubernetes.io/projected/83b9efdd-4a34-428e-9f19-3ad6be585fe0-kube-api-access-xm99z\") pod \"83b9efdd-4a34-428e-9f19-3ad6be585fe0\" (UID: \"83b9efdd-4a34-428e-9f19-3ad6be585fe0\") "
Oct 30 04:29:33.683260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:33.677647    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83b9efdd-4a34-428e-9f19-3ad6be585fe0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83b9efdd-4a34-428e-9f19-3ad6be585fe0-plugins-config-volume\") pod \"83b9efdd-4a34-428e-9f19-3ad6be585fe0\" (UID: \"83b9efdd-4a34-428e-9f19-3ad6be585fe0\") "
Oct 30 04:29:33.689809 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:33.672872    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b9efdd-4a34-428e-9f19-3ad6be585fe0-kube-api-access-xm99z" pod "83b9efdd-4a34-428e-9f19-3ad6be585fe0" (UID: "83b9efdd-4a34-428e-9f19-3ad6be585fe0"). InnerVolumeSpecName "kube-api-access-xm99z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:29:33.689954 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:33.688859    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83b9efdd-4a34-428e-9f19-3ad6be585fe0-plugins-config-volume" pod "83b9efdd-4a34-428e-9f19-3ad6be585fe0" (UID: "83b9efdd-4a34-428e-9f19-3ad6be585fe0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:29:33.769942 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:33.769864    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q"
Oct 30 04:29:33.790704 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:33.790673    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm99z\" (UniqueName: \"kubernetes.io/projected/83b9efdd-4a34-428e-9f19-3ad6be585fe0-kube-api-access-xm99z\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:33.790810 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:33.790706    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83b9efdd-4a34-428e-9f19-3ad6be585fe0-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:33.794065 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:33.794040    4874 generic.go:356] "Generic (PLEG): container finished" podID="83b9efdd-4a34-428e-9f19-3ad6be585fe0" containerID="55aa0b0bd26227f078b51eaad0ce87dedc9bb0ef982bf092e3b4947f146ddeae" exitCode=0
Oct 30 04:29:33.794153 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:33.794078    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f"
Oct 30 04:29:33.794207 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:33.794080    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f" event={"ID":"83b9efdd-4a34-428e-9f19-3ad6be585fe0","Type":"ContainerDied","Data":"55aa0b0bd26227f078b51eaad0ce87dedc9bb0ef982bf092e3b4947f146ddeae"}
Oct 30 04:29:33.794257 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:33.794225    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f" event={"ID":"83b9efdd-4a34-428e-9f19-3ad6be585fe0","Type":"ContainerDied","Data":"466e07587cfcd923bf15c4ee596cbb2830c6966dfdf0f42c7669fab3fd6250f7"}
Oct 30 04:29:33.794294 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:33.794257    4874 scope.go:122] "RemoveContainer" containerID="55aa0b0bd26227f078b51eaad0ce87dedc9bb0ef982bf092e3b4947f146ddeae"
Oct 30 04:29:33.803628 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:33.803614    4874 scope.go:122] "RemoveContainer" containerID="55aa0b0bd26227f078b51eaad0ce87dedc9bb0ef982bf092e3b4947f146ddeae"
Oct 30 04:29:33.803894 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:29:33.803876    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55aa0b0bd26227f078b51eaad0ce87dedc9bb0ef982bf092e3b4947f146ddeae\": container with ID starting with 55aa0b0bd26227f078b51eaad0ce87dedc9bb0ef982bf092e3b4947f146ddeae not found: ID does not exist" containerID="55aa0b0bd26227f078b51eaad0ce87dedc9bb0ef982bf092e3b4947f146ddeae"
Oct 30 04:29:33.803934 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:33.803902    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55aa0b0bd26227f078b51eaad0ce87dedc9bb0ef982bf092e3b4947f146ddeae"} err="failed to get container status \"55aa0b0bd26227f078b51eaad0ce87dedc9bb0ef982bf092e3b4947f146ddeae\": rpc error: code = NotFound desc = could not find container \"55aa0b0bd26227f078b51eaad0ce87dedc9bb0ef982bf092e3b4947f146ddeae\": container with ID starting with 55aa0b0bd26227f078b51eaad0ce87dedc9bb0ef982bf092e3b4947f146ddeae not found: ID does not exist"
Oct 30 04:29:33.852580 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:33.852550    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f"]
Oct 30 04:29:33.860193 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:33.860167    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-baa474-844c576744-t2w9f"]
Oct 30 04:29:34.729051 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:34.729014    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83b9efdd-4a34-428e-9f19-3ad6be585fe0" path="/var/lib/kubelet/pods/83b9efdd-4a34-428e-9f19-3ad6be585fe0/volumes"
Oct 30 04:29:37.501217 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:37.501182    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"]
Oct 30 04:29:37.501709 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:37.501456    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t" podUID="30b33bf9-b5cb-473c-b6a4-d30dd097fd72" containerName="istio-proxy" containerID="cri-o://79befbb93d75436cb8048d2a297155598543a371632244b81b30319d1d8cecba" gracePeriod=30
Oct 30 04:29:38.467528 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:38.467491    4874 patch_prober.go:29] interesting pod/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 04:29:38.467815 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:38.467778    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t" podUID="30b33bf9-b5cb-473c-b6a4-d30dd097fd72" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:29:41.785262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:41.782867    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:29:42.745810 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.745779    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v"]
Oct 30 04:29:42.746227 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.746179    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v" podUID="e6201c5f-2acd-4e8e-8fc6-63a7445369cb" containerName="payload-pre-processing" containerID="cri-o://96fd018b62c0dba2da99e5f5799077fa07e0ffc89aa3731e201e0ebe88203ac0" gracePeriod=30
Oct 30 04:29:42.778576 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.778505    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:29:42.877544 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.877512    4874 generic.go:356] "Generic (PLEG): container finished" podID="30b33bf9-b5cb-473c-b6a4-d30dd097fd72" containerID="79befbb93d75436cb8048d2a297155598543a371632244b81b30319d1d8cecba" exitCode=0
Oct 30 04:29:42.878289 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.877579    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"
Oct 30 04:29:42.878289 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.877592    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t" event={"ID":"30b33bf9-b5cb-473c-b6a4-d30dd097fd72","Type":"ContainerDied","Data":"79befbb93d75436cb8048d2a297155598543a371632244b81b30319d1d8cecba"}
Oct 30 04:29:42.878475 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.878329    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t" event={"ID":"30b33bf9-b5cb-473c-b6a4-d30dd097fd72","Type":"ContainerDied","Data":"736692151e421b8190dc4f4ac4ea65addd2c86e2d2e159a992297d4838b18fa8"}
Oct 30 04:29:42.878475 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.878382    4874 scope.go:122] "RemoveContainer" containerID="79befbb93d75436cb8048d2a297155598543a371632244b81b30319d1d8cecba"
Oct 30 04:29:42.883110 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.883081    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-workload-certs\") pod \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") "
Oct 30 04:29:42.883253 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.883165    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-token\" (UniqueName: \"kubernetes.io/projected/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-token\") pod \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") "
Oct 30 04:29:42.883253 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.883211    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-envoy\") pod \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") "
Oct 30 04:29:42.883374 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.883351    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-data\") pod \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") "
Oct 30 04:29:42.883425 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.883387    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-credential-socket\") pod \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") "
Oct 30 04:29:42.883425 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.883419    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-podinfo\") pod \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") "
Oct 30 04:29:42.883519 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.883463    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istiod-ca-cert\") pod \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") "
Oct 30 04:29:42.883580 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.883560    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-kube-api-access-8llsq\" (UniqueName: \"kubernetes.io/projected/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-kube-api-access-8llsq\") pod \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") "
Oct 30 04:29:42.883648 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.883610    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-workload-socket\") pod \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\" (UID: \"30b33bf9-b5cb-473c-b6a4-d30dd097fd72\") "
Oct 30 04:29:42.884408 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.884073    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-credential-socket" pod "30b33bf9-b5cb-473c-b6a4-d30dd097fd72" (UID: "30b33bf9-b5cb-473c-b6a4-d30dd097fd72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:29:42.884408 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.884219    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istiod-ca-cert" pod "30b33bf9-b5cb-473c-b6a4-d30dd097fd72" (UID: "30b33bf9-b5cb-473c-b6a4-d30dd097fd72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:29:42.884408 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.884333    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-data" pod "30b33bf9-b5cb-473c-b6a4-d30dd097fd72" (UID: "30b33bf9-b5cb-473c-b6a4-d30dd097fd72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:29:42.884606 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.884590    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-workload-certs" pod "30b33bf9-b5cb-473c-b6a4-d30dd097fd72" (UID: "30b33bf9-b5cb-473c-b6a4-d30dd097fd72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:29:42.885967 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.885938    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-workload-socket" pod "30b33bf9-b5cb-473c-b6a4-d30dd097fd72" (UID: "30b33bf9-b5cb-473c-b6a4-d30dd097fd72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:29:42.886692 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.886608    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-podinfo" pod "30b33bf9-b5cb-473c-b6a4-d30dd097fd72" (UID: "30b33bf9-b5cb-473c-b6a4-d30dd097fd72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:29:42.886692 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.886620    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-kube-api-access-8llsq" pod "30b33bf9-b5cb-473c-b6a4-d30dd097fd72" (UID: "30b33bf9-b5cb-473c-b6a4-d30dd097fd72"). InnerVolumeSpecName "kube-api-access-8llsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:29:42.886692 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.886674    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-envoy" pod "30b33bf9-b5cb-473c-b6a4-d30dd097fd72" (UID: "30b33bf9-b5cb-473c-b6a4-d30dd097fd72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:29:42.887098 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.887072    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-token" pod "30b33bf9-b5cb-473c-b6a4-d30dd097fd72" (UID: "30b33bf9-b5cb-473c-b6a4-d30dd097fd72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:29:42.891576 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.890453    4874 scope.go:122] "RemoveContainer" containerID="67fe80662a55be354116d444a12147776a684fa5f7b0911fa6a40e67101995a2"
Oct 30 04:29:42.914407 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.914380    4874 scope.go:122] "RemoveContainer" containerID="79befbb93d75436cb8048d2a297155598543a371632244b81b30319d1d8cecba"
Oct 30 04:29:42.914790 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:29:42.914759    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79befbb93d75436cb8048d2a297155598543a371632244b81b30319d1d8cecba\": container with ID starting with 79befbb93d75436cb8048d2a297155598543a371632244b81b30319d1d8cecba not found: ID does not exist" containerID="79befbb93d75436cb8048d2a297155598543a371632244b81b30319d1d8cecba"
Oct 30 04:29:42.914865 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.914802    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79befbb93d75436cb8048d2a297155598543a371632244b81b30319d1d8cecba"} err="failed to get container status \"79befbb93d75436cb8048d2a297155598543a371632244b81b30319d1d8cecba\": rpc error: code = NotFound desc = could not find container \"79befbb93d75436cb8048d2a297155598543a371632244b81b30319d1d8cecba\": container with ID starting with 79befbb93d75436cb8048d2a297155598543a371632244b81b30319d1d8cecba not found: ID does not exist"
Oct 30 04:29:42.914865 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.914827    4874 scope.go:122] "RemoveContainer" containerID="67fe80662a55be354116d444a12147776a684fa5f7b0911fa6a40e67101995a2"
Oct 30 04:29:42.915168 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:29:42.915145    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67fe80662a55be354116d444a12147776a684fa5f7b0911fa6a40e67101995a2\": container with ID starting with 67fe80662a55be354116d444a12147776a684fa5f7b0911fa6a40e67101995a2 not found: ID does not exist" containerID="67fe80662a55be354116d444a12147776a684fa5f7b0911fa6a40e67101995a2"
Oct 30 04:29:42.915252 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.915179    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67fe80662a55be354116d444a12147776a684fa5f7b0911fa6a40e67101995a2"} err="failed to get container status \"67fe80662a55be354116d444a12147776a684fa5f7b0911fa6a40e67101995a2\": rpc error: code = NotFound desc = could not find container \"67fe80662a55be354116d444a12147776a684fa5f7b0911fa6a40e67101995a2\": container with ID starting with 67fe80662a55be354116d444a12147776a684fa5f7b0911fa6a40e67101995a2 not found: ID does not exist"
Oct 30 04:29:42.985156 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.985129    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8llsq\" (UniqueName: \"kubernetes.io/projected/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-kube-api-access-8llsq\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:42.985156 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.985156    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:42.985356 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.985167    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:42.985356 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.985177    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:42.985356 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.985185    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:42.985356 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.985193    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:42.985356 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.985200    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:42.985356 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.985209    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:42.985356 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:42.985218    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b33bf9-b5cb-473c-b6a4-d30dd097fd72-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:43.053218 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:43.053195    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v"
Oct 30 04:29:43.198153 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:43.198116    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6201c5f-2acd-4e8e-8fc6-63a7445369cb-kube-api-access-lfxlm\" (UniqueName: \"kubernetes.io/projected/e6201c5f-2acd-4e8e-8fc6-63a7445369cb-kube-api-access-lfxlm\") pod \"e6201c5f-2acd-4e8e-8fc6-63a7445369cb\" (UID: \"e6201c5f-2acd-4e8e-8fc6-63a7445369cb\") "
Oct 30 04:29:43.198385 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:43.198288    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6201c5f-2acd-4e8e-8fc6-63a7445369cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6201c5f-2acd-4e8e-8fc6-63a7445369cb-plugins-config-volume\") pod \"e6201c5f-2acd-4e8e-8fc6-63a7445369cb\" (UID: \"e6201c5f-2acd-4e8e-8fc6-63a7445369cb\") "
Oct 30 04:29:43.199360 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:43.199329    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6201c5f-2acd-4e8e-8fc6-63a7445369cb-plugins-config-volume" pod "e6201c5f-2acd-4e8e-8fc6-63a7445369cb" (UID: "e6201c5f-2acd-4e8e-8fc6-63a7445369cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:29:43.210660 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:43.210593    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6201c5f-2acd-4e8e-8fc6-63a7445369cb-kube-api-access-lfxlm" pod "e6201c5f-2acd-4e8e-8fc6-63a7445369cb" (UID: "e6201c5f-2acd-4e8e-8fc6-63a7445369cb"). InnerVolumeSpecName "kube-api-access-lfxlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:29:43.250871 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:43.250827    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"]
Oct 30 04:29:43.257748 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:43.257720    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6dcea5-openshift-default-5848589cc5-xfd5t"]
Oct 30 04:29:43.299890 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:43.299768    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6201c5f-2acd-4e8e-8fc6-63a7445369cb-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:43.299890 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:43.299805    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfxlm\" (UniqueName: \"kubernetes.io/projected/e6201c5f-2acd-4e8e-8fc6-63a7445369cb-kube-api-access-lfxlm\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:43.885571 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:43.885532    4874 generic.go:356] "Generic (PLEG): container finished" podID="e6201c5f-2acd-4e8e-8fc6-63a7445369cb" containerID="96fd018b62c0dba2da99e5f5799077fa07e0ffc89aa3731e201e0ebe88203ac0" exitCode=0
Oct 30 04:29:43.886009 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:43.885577    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v"
Oct 30 04:29:43.886009 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:43.885605    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v" event={"ID":"e6201c5f-2acd-4e8e-8fc6-63a7445369cb","Type":"ContainerDied","Data":"96fd018b62c0dba2da99e5f5799077fa07e0ffc89aa3731e201e0ebe88203ac0"}
Oct 30 04:29:43.886009 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:43.885656    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v" event={"ID":"e6201c5f-2acd-4e8e-8fc6-63a7445369cb","Type":"ContainerDied","Data":"cee5a71f82880a33f6407b5828f6f2b34b2126de17b8cd6fd684602b1b7d5d68"}
Oct 30 04:29:43.886009 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:43.885679    4874 scope.go:122] "RemoveContainer" containerID="96fd018b62c0dba2da99e5f5799077fa07e0ffc89aa3731e201e0ebe88203ac0"
Oct 30 04:29:43.894888 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:43.894870    4874 scope.go:122] "RemoveContainer" containerID="96fd018b62c0dba2da99e5f5799077fa07e0ffc89aa3731e201e0ebe88203ac0"
Oct 30 04:29:43.895146 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:29:43.895128    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96fd018b62c0dba2da99e5f5799077fa07e0ffc89aa3731e201e0ebe88203ac0\": container with ID starting with 96fd018b62c0dba2da99e5f5799077fa07e0ffc89aa3731e201e0ebe88203ac0 not found: ID does not exist" containerID="96fd018b62c0dba2da99e5f5799077fa07e0ffc89aa3731e201e0ebe88203ac0"
Oct 30 04:29:43.895187 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:43.895153    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96fd018b62c0dba2da99e5f5799077fa07e0ffc89aa3731e201e0ebe88203ac0"} err="failed to get container status \"96fd018b62c0dba2da99e5f5799077fa07e0ffc89aa3731e201e0ebe88203ac0\": rpc error: code = NotFound desc = could not find container \"96fd018b62c0dba2da99e5f5799077fa07e0ffc89aa3731e201e0ebe88203ac0\": container with ID starting with 96fd018b62c0dba2da99e5f5799077fa07e0ffc89aa3731e201e0ebe88203ac0 not found: ID does not exist"
Oct 30 04:29:43.934558 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:43.934534    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v"]
Oct 30 04:29:43.939887 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:43.939867    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-5f75c99dc-vsv9v"]
Oct 30 04:29:44.728730 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:44.728695    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30b33bf9-b5cb-473c-b6a4-d30dd097fd72" path="/var/lib/kubelet/pods/30b33bf9-b5cb-473c-b6a4-d30dd097fd72/volumes"
Oct 30 04:29:44.729173 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:44.729151    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6201c5f-2acd-4e8e-8fc6-63a7445369cb" path="/var/lib/kubelet/pods/e6201c5f-2acd-4e8e-8fc6-63a7445369cb/volumes"
Oct 30 04:29:48.983683 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:48.983643    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw"]
Oct 30 04:29:48.984288 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:48.983884    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw" podUID="30954ce5-9300-4778-8322-172239088b73" containerName="payload-processing" containerID="cri-o://df482d307a8d850ff4f0072bf07da1042829abaf52d49e6aa63d820565e10793" gracePeriod=30
Oct 30 04:29:49.010770 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.010740    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw"]
Oct 30 04:29:49.011168 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.011137    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw" podUID="b03379d8-4074-4c65-b918-77ed905261a2" containerName="payload-pre-processing" containerID="cri-o://67c554962b7daf8a509436bab474381931c1874ede20b7f528ef5bdc1d431a45" gracePeriod=30
Oct 30 04:29:49.274550 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.274517    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw"
Oct 30 04:29:49.373306 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.365552    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw"
Oct 30 04:29:49.386900 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.386860    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30954ce5-9300-4778-8322-172239088b73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30954ce5-9300-4778-8322-172239088b73-plugins-config-volume\") pod \"30954ce5-9300-4778-8322-172239088b73\" (UID: \"30954ce5-9300-4778-8322-172239088b73\") "
Oct 30 04:29:49.387058 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.386991    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b03379d8-4074-4c65-b918-77ed905261a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b03379d8-4074-4c65-b918-77ed905261a2-plugins-config-volume\") pod \"b03379d8-4074-4c65-b918-77ed905261a2\" (UID: \"b03379d8-4074-4c65-b918-77ed905261a2\") "
Oct 30 04:29:49.387100 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.387086    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b03379d8-4074-4c65-b918-77ed905261a2-kube-api-access-wsvmx\" (UniqueName: \"kubernetes.io/projected/b03379d8-4074-4c65-b918-77ed905261a2-kube-api-access-wsvmx\") pod \"b03379d8-4074-4c65-b918-77ed905261a2\" (UID: \"b03379d8-4074-4c65-b918-77ed905261a2\") "
Oct 30 04:29:49.387139 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.387125    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30954ce5-9300-4778-8322-172239088b73-kube-api-access-fqb7b\" (UniqueName: \"kubernetes.io/projected/30954ce5-9300-4778-8322-172239088b73-kube-api-access-fqb7b\") pod \"30954ce5-9300-4778-8322-172239088b73\" (UID: \"30954ce5-9300-4778-8322-172239088b73\") "
Oct 30 04:29:49.388308 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.388275    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b03379d8-4074-4c65-b918-77ed905261a2-plugins-config-volume" pod "b03379d8-4074-4c65-b918-77ed905261a2" (UID: "b03379d8-4074-4c65-b918-77ed905261a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:29:49.388786 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.388762    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30954ce5-9300-4778-8322-172239088b73-plugins-config-volume" pod "30954ce5-9300-4778-8322-172239088b73" (UID: "30954ce5-9300-4778-8322-172239088b73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:29:49.394286 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.393510    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03379d8-4074-4c65-b918-77ed905261a2-kube-api-access-wsvmx" pod "b03379d8-4074-4c65-b918-77ed905261a2" (UID: "b03379d8-4074-4c65-b918-77ed905261a2"). InnerVolumeSpecName "kube-api-access-wsvmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:29:49.395763 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.395733    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30954ce5-9300-4778-8322-172239088b73-kube-api-access-fqb7b" pod "30954ce5-9300-4778-8322-172239088b73" (UID: "30954ce5-9300-4778-8322-172239088b73"). InnerVolumeSpecName "kube-api-access-fqb7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:29:49.488883 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.488850    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsvmx\" (UniqueName: \"kubernetes.io/projected/b03379d8-4074-4c65-b918-77ed905261a2-kube-api-access-wsvmx\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:49.488883 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.488877    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqb7b\" (UniqueName: \"kubernetes.io/projected/30954ce5-9300-4778-8322-172239088b73-kube-api-access-fqb7b\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:49.488883 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.488889    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30954ce5-9300-4778-8322-172239088b73-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:49.489115 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.488899    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b03379d8-4074-4c65-b918-77ed905261a2-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:49.969142 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.969105    4874 generic.go:356] "Generic (PLEG): container finished" podID="b03379d8-4074-4c65-b918-77ed905261a2" containerID="67c554962b7daf8a509436bab474381931c1874ede20b7f528ef5bdc1d431a45" exitCode=0
Oct 30 04:29:49.969344 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.969145    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw"
Oct 30 04:29:49.969344 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.969191    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw" event={"ID":"b03379d8-4074-4c65-b918-77ed905261a2","Type":"ContainerDied","Data":"67c554962b7daf8a509436bab474381931c1874ede20b7f528ef5bdc1d431a45"}
Oct 30 04:29:49.969344 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.969265    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw" event={"ID":"b03379d8-4074-4c65-b918-77ed905261a2","Type":"ContainerDied","Data":"89401ddfdb31ccfe312637ff490c325fe18214f3271fdbac2bcc0e3cb3a83d77"}
Oct 30 04:29:49.969344 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.969294    4874 scope.go:122] "RemoveContainer" containerID="67c554962b7daf8a509436bab474381931c1874ede20b7f528ef5bdc1d431a45"
Oct 30 04:29:49.971917 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.971889    4874 generic.go:356] "Generic (PLEG): container finished" podID="30954ce5-9300-4778-8322-172239088b73" containerID="df482d307a8d850ff4f0072bf07da1042829abaf52d49e6aa63d820565e10793" exitCode=0
Oct 30 04:29:49.972055 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.971919    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw"
Oct 30 04:29:49.972055 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.971971    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw" event={"ID":"30954ce5-9300-4778-8322-172239088b73","Type":"ContainerDied","Data":"df482d307a8d850ff4f0072bf07da1042829abaf52d49e6aa63d820565e10793"}
Oct 30 04:29:49.972055 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.972021    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw" event={"ID":"30954ce5-9300-4778-8322-172239088b73","Type":"ContainerDied","Data":"6429eba1cce4ce856e05eab4d654aef3828c48f9c2960806547078308832cd5c"}
Oct 30 04:29:49.981053 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.981036    4874 scope.go:122] "RemoveContainer" containerID="67c554962b7daf8a509436bab474381931c1874ede20b7f528ef5bdc1d431a45"
Oct 30 04:29:49.981381 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:29:49.981363    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67c554962b7daf8a509436bab474381931c1874ede20b7f528ef5bdc1d431a45\": container with ID starting with 67c554962b7daf8a509436bab474381931c1874ede20b7f528ef5bdc1d431a45 not found: ID does not exist" containerID="67c554962b7daf8a509436bab474381931c1874ede20b7f528ef5bdc1d431a45"
Oct 30 04:29:49.981463 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.981389    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67c554962b7daf8a509436bab474381931c1874ede20b7f528ef5bdc1d431a45"} err="failed to get container status \"67c554962b7daf8a509436bab474381931c1874ede20b7f528ef5bdc1d431a45\": rpc error: code = NotFound desc = could not find container \"67c554962b7daf8a509436bab474381931c1874ede20b7f528ef5bdc1d431a45\": container with ID starting with 67c554962b7daf8a509436bab474381931c1874ede20b7f528ef5bdc1d431a45 not found: ID does not exist"
Oct 30 04:29:49.981463 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.981405    4874 scope.go:122] "RemoveContainer" containerID="df482d307a8d850ff4f0072bf07da1042829abaf52d49e6aa63d820565e10793"
Oct 30 04:29:49.992030 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.991851    4874 scope.go:122] "RemoveContainer" containerID="df482d307a8d850ff4f0072bf07da1042829abaf52d49e6aa63d820565e10793"
Oct 30 04:29:49.992277 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:29:49.992133    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df482d307a8d850ff4f0072bf07da1042829abaf52d49e6aa63d820565e10793\": container with ID starting with df482d307a8d850ff4f0072bf07da1042829abaf52d49e6aa63d820565e10793 not found: ID does not exist" containerID="df482d307a8d850ff4f0072bf07da1042829abaf52d49e6aa63d820565e10793"
Oct 30 04:29:49.992277 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:49.992159    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df482d307a8d850ff4f0072bf07da1042829abaf52d49e6aa63d820565e10793"} err="failed to get container status \"df482d307a8d850ff4f0072bf07da1042829abaf52d49e6aa63d820565e10793\": rpc error: code = NotFound desc = could not find container \"df482d307a8d850ff4f0072bf07da1042829abaf52d49e6aa63d820565e10793\": container with ID starting with df482d307a8d850ff4f0072bf07da1042829abaf52d49e6aa63d820565e10793 not found: ID does not exist"
Oct 30 04:29:50.039163 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.039134    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw"]
Oct 30 04:29:50.047720 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.047693    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-5554cbc85f-7lvkw"]
Oct 30 04:29:50.095154 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.095128    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw"]
Oct 30 04:29:50.104699 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.104671    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b1a417-7479df4cc6-7jwmw"]
Oct 30 04:29:50.190036 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.190005    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"]
Oct 30 04:29:50.190758 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.190732    4874 cpu_manager.go:414] "containerMap: removing container" podUID="30b33bf9-b5cb-473c-b6a4-d30dd097fd72" containerName="istio-proxy"
Oct 30 04:29:50.190758 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.190756    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30b33bf9-b5cb-473c-b6a4-d30dd097fd72" containerName="istio-proxy"
Oct 30 04:29:50.190964 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.190767    4874 cpu_manager.go:414] "containerMap: removing container" podUID="30954ce5-9300-4778-8322-172239088b73" containerName="payload-processing"
Oct 30 04:29:50.190964 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.190776    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30954ce5-9300-4778-8322-172239088b73" containerName="payload-processing"
Oct 30 04:29:50.190964 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.190798    4874 cpu_manager.go:414] "containerMap: removing container" podUID="b03379d8-4074-4c65-b918-77ed905261a2" containerName="payload-pre-processing"
Oct 30 04:29:50.190964 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.190805    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b03379d8-4074-4c65-b918-77ed905261a2" containerName="payload-pre-processing"
Oct 30 04:29:50.190964 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.190849    4874 cpu_manager.go:414] "containerMap: removing container" podUID="e6201c5f-2acd-4e8e-8fc6-63a7445369cb" containerName="payload-pre-processing"
Oct 30 04:29:50.190964 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.190856    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6201c5f-2acd-4e8e-8fc6-63a7445369cb" containerName="payload-pre-processing"
Oct 30 04:29:50.190964 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.190870    4874 cpu_manager.go:414] "containerMap: removing container" podUID="9c67bdeb-196a-4314-a541-48cbac967a46" containerName="maas-api"
Oct 30 04:29:50.190964 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.190876    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c67bdeb-196a-4314-a541-48cbac967a46" containerName="maas-api"
Oct 30 04:29:50.190964 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.190885    4874 cpu_manager.go:414] "containerMap: removing container" podUID="83b9efdd-4a34-428e-9f19-3ad6be585fe0" containerName="payload-processing"
Oct 30 04:29:50.190964 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.190894    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b9efdd-4a34-428e-9f19-3ad6be585fe0" containerName="payload-processing"
Oct 30 04:29:50.191537 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.190999    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c67bdeb-196a-4314-a541-48cbac967a46" containerName="maas-api"
Oct 30 04:29:50.191537 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.191012    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b9efdd-4a34-428e-9f19-3ad6be585fe0" containerName="payload-processing"
Oct 30 04:29:50.191537 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.191022    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="30b33bf9-b5cb-473c-b6a4-d30dd097fd72" containerName="istio-proxy"
Oct 30 04:29:50.191537 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.191031    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="30954ce5-9300-4778-8322-172239088b73" containerName="payload-processing"
Oct 30 04:29:50.191537 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.191042    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="b03379d8-4074-4c65-b918-77ed905261a2" containerName="payload-pre-processing"
Oct 30 04:29:50.191537 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.191057    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6201c5f-2acd-4e8e-8fc6-63a7445369cb" containerName="payload-pre-processing"
Oct 30 04:29:50.195830 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.195786    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.212042 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.212016    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-dbabed-openshift-default-dockercfg-ljsh9\""
Oct 30 04:29:50.225450 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.225428    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"]
Oct 30 04:29:50.301049 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.300464    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc662\" (UniqueName: \"kubernetes.io/projected/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-kube-api-access-wc662\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.301049 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.300543    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istiod-ca-cert\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.301049 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.300594    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-workload-socket\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.301049 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.300628    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-token\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.301049 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.300663    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-workload-certs\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.301049 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.300716    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-podinfo\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.301049 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.300777    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-credential-socket\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.301049 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.300829    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-data\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.301049 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.300866    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-envoy\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.406267 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.404934    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istiod-ca-cert\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.406267 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.405010    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-workload-socket\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.406267 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.405048    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-token\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.406267 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.405082    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-workload-certs\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.406267 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.405134    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-podinfo\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.406267 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.405194    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-credential-socket\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.406267 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.405268    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-data\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.406267 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.405320    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-envoy\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.406267 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.405397    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc662\" (UniqueName: \"kubernetes.io/projected/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-kube-api-access-wc662\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.406844 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.406537    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istiod-ca-cert\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.406844 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.406830    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-workload-socket\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.410267 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.407343    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-workload-certs\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.418260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.417802    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-data\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.418260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.418196    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-credential-socket\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.430653 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.423300    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-envoy\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.430653 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.425429    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-podinfo\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.440889 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.439959    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc662\" (UniqueName: \"kubernetes.io/projected/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-kube-api-access-wc662\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.441458 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.441433    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-token\") pod \"e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") " pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.516213 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.516133    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:50.810756 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.804154    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30954ce5-9300-4778-8322-172239088b73" path="/var/lib/kubelet/pods/30954ce5-9300-4778-8322-172239088b73/volumes"
Oct 30 04:29:50.810756 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.804782    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b03379d8-4074-4c65-b918-77ed905261a2" path="/var/lib/kubelet/pods/b03379d8-4074-4c65-b918-77ed905261a2/volumes"
Oct 30 04:29:50.846407 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.846281    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"]
Oct 30 04:29:50.849840 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:29:50.849811    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a69deca_d8cf_4ea4_bb1f_15bee6a789a4.slice/crio-06014e139cbf0dad704200631fc5f52f4bbb15d729f008c4950720e2895ccf10 WatchSource:0}: Error finding container 06014e139cbf0dad704200631fc5f52f4bbb15d729f008c4950720e2895ccf10: Status 404 returned error can't find the container with id 06014e139cbf0dad704200631fc5f52f4bbb15d729f008c4950720e2895ccf10
Oct 30 04:29:50.852299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.852265    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:29:50.852386 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.852337    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:29:50.852386 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.852368    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:29:50.980987 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.980937    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s" event={"ID":"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4","Type":"ContainerStarted","Data":"220354a581ef639ab086606c6b464cc89579d421dae2d836bc820fd532dc51a8"}
Oct 30 04:29:50.981159 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:50.981003    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s" event={"ID":"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4","Type":"ContainerStarted","Data":"06014e139cbf0dad704200631fc5f52f4bbb15d729f008c4950720e2895ccf10"}
Oct 30 04:29:51.039272 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:51.039207    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s" podStartSLOduration=1.039192783 podStartE2EDuration="1.039192783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:29:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:29:51.037883512 +0000 UTC m=+1910.450279084" watchObservedRunningTime="2026-10-30 04:29:51.039192783 +0000 UTC m=+1910.451588353"
Oct 30 04:29:51.517258 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:51.517208    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:51.522881 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:51.522851    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:51.989389 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:51.989360    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:51.990391 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:51.990368    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:29:53.181089 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:53.181053    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"]
Oct 30 04:29:53.181623 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:53.181308    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz" podUID="ba8e0c94-8f6b-40dd-86f9-3973177dd696" containerName="istio-proxy" containerID="cri-o://ab03c3db0059bcc14c41937f07627fce74e16ef0ee6ad0a9df95c583d0405cff" gracePeriod=30
Oct 30 04:29:55.610362 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:29:55.610324    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:29:56.136807 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:56.134316    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9"]
Oct 30 04:29:56.136807 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:56.135095    4874 cpu_manager.go:414] "containerMap: removing container" podUID="30b33bf9-b5cb-473c-b6a4-d30dd097fd72" containerName="istio-proxy"
Oct 30 04:29:56.136807 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:56.135116    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30b33bf9-b5cb-473c-b6a4-d30dd097fd72" containerName="istio-proxy"
Oct 30 04:29:56.136807 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:56.135290    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="30b33bf9-b5cb-473c-b6a4-d30dd097fd72" containerName="istio-proxy"
Oct 30 04:29:56.151960 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:56.138984    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9"
Oct 30 04:29:56.177920 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:56.177891    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9"]
Oct 30 04:29:56.194261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:56.183388    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-dbabed-dockercfg-vv5vr\""
Oct 30 04:29:56.194261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:56.183446    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-dbabed\""
Oct 30 04:29:56.206622 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:56.206583    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd33009-bafc-4f73-a476-f5d852966202-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9\" (UID: \"ecd33009-bafc-4f73-a476-f5d852966202\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9"
Oct 30 04:29:56.206973 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:56.206949    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6bdm\" (UniqueName: \"kubernetes.io/projected/ecd33009-bafc-4f73-a476-f5d852966202-kube-api-access-t6bdm\") pod \"payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9\" (UID: \"ecd33009-bafc-4f73-a476-f5d852966202\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9"
Oct 30 04:29:56.307968 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:56.307933    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd33009-bafc-4f73-a476-f5d852966202-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9\" (UID: \"ecd33009-bafc-4f73-a476-f5d852966202\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9"
Oct 30 04:29:56.308131 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:56.308001    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6bdm\" (UniqueName: \"kubernetes.io/projected/ecd33009-bafc-4f73-a476-f5d852966202-kube-api-access-t6bdm\") pod \"payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9\" (UID: \"ecd33009-bafc-4f73-a476-f5d852966202\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9"
Oct 30 04:29:56.308607 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:56.308589    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd33009-bafc-4f73-a476-f5d852966202-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9\" (UID: \"ecd33009-bafc-4f73-a476-f5d852966202\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9"
Oct 30 04:29:56.332258 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:56.332223    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6bdm\" (UniqueName: \"kubernetes.io/projected/ecd33009-bafc-4f73-a476-f5d852966202-kube-api-access-t6bdm\") pod \"payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9\" (UID: \"ecd33009-bafc-4f73-a476-f5d852966202\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9"
Oct 30 04:29:56.479616 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:56.479589    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9"
Oct 30 04:29:56.785011 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:56.784986    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9"]
Oct 30 04:29:56.787416 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:29:56.787383    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd33009_bafc_4f73_a476_f5d852966202.slice/crio-26ec6bbf42ab11c94e143537b5c5ae4c9ca2314dd0da6a6755ef62941dc3f709 WatchSource:0}: Error finding container 26ec6bbf42ab11c94e143537b5c5ae4c9ca2314dd0da6a6755ef62941dc3f709: Status 404 returned error can't find the container with id 26ec6bbf42ab11c94e143537b5c5ae4c9ca2314dd0da6a6755ef62941dc3f709
Oct 30 04:29:57.029429 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:57.029329    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9" event={"ID":"ecd33009-bafc-4f73-a476-f5d852966202","Type":"ContainerStarted","Data":"4df9644768e08626e7827367fec3f53c93443484c81d1a5df48c0c57773da15f"}
Oct 30 04:29:57.029624 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:57.029543    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9"
Oct 30 04:29:57.029624 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:57.029572    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9" event={"ID":"ecd33009-bafc-4f73-a476-f5d852966202","Type":"ContainerStarted","Data":"26ec6bbf42ab11c94e143537b5c5ae4c9ca2314dd0da6a6755ef62941dc3f709"}
Oct 30 04:29:57.082610 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:57.082513    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9" podStartSLOduration=1.082500283 podStartE2EDuration="1.082500283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:29:57.081691515 +0000 UTC m=+1916.494087087" watchObservedRunningTime="2026-10-30 04:29:57.082500283 +0000 UTC m=+1916.494895869"
Oct 30 04:29:57.807683 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:57.807654    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2"]
Oct 30 04:29:57.808177 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:57.807889    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2" podUID="d1950c75-bacc-4f16-91f0-66e2f6b3cf18" containerName="maas-api" containerID="cri-o://cea7bc32848dcb9f0ee5de150302cf6407d373338ff7661747152d052bba5c21" gracePeriod=30
Oct 30 04:29:57.865827 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:57.865792    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n"]
Oct 30 04:29:57.870066 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:57.870044    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n"
Oct 30 04:29:57.890292 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:57.889493    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-58cdc8-dockercfg-vdk94\""
Oct 30 04:29:57.890292 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:57.889546    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-58cdc8\""
Oct 30 04:29:57.890292 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:57.890030    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n"]
Oct 30 04:29:57.934224 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:57.934192    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9prc6\" (UniqueName: \"kubernetes.io/projected/1d4d2573-179f-46bc-be32-e3dcc140437f-kube-api-access-9prc6\") pod \"payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n\" (UID: \"1d4d2573-179f-46bc-be32-e3dcc140437f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n"
Oct 30 04:29:57.934224 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:57.934387    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d4d2573-179f-46bc-be32-e3dcc140437f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n\" (UID: \"1d4d2573-179f-46bc-be32-e3dcc140437f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n"
Oct 30 04:29:57.991101 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:57.991071    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk"]
Oct 30 04:29:57.991775 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:57.991336    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk" podUID="d128d0ee-3fa4-4023-9333-cb7fa4ff5940" containerName="payload-processing" containerID="cri-o://16cc713a89d924e1831ee15a27825a562ada950d42b8ded553dc5bc50e141c94" gracePeriod=30
Oct 30 04:29:58.017500 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.017472    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz"]
Oct 30 04:29:58.017742 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.017716    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz" podUID="54704513-6246-41c9-978f-77b8d99e085b" containerName="payload-pre-processing" containerID="cri-o://de1a7fed98e7620a237d9b0f21072f7c2fa3f76afbdb4366a3a37c1e81449a19" gracePeriod=30
Oct 30 04:29:58.035666 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.035633    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9prc6\" (UniqueName: \"kubernetes.io/projected/1d4d2573-179f-46bc-be32-e3dcc140437f-kube-api-access-9prc6\") pod \"payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n\" (UID: \"1d4d2573-179f-46bc-be32-e3dcc140437f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n"
Oct 30 04:29:58.035832 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.035812    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d4d2573-179f-46bc-be32-e3dcc140437f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n\" (UID: \"1d4d2573-179f-46bc-be32-e3dcc140437f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n"
Oct 30 04:29:58.036877 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.036848    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d4d2573-179f-46bc-be32-e3dcc140437f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n\" (UID: \"1d4d2573-179f-46bc-be32-e3dcc140437f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n"
Oct 30 04:29:58.041218 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.041191    4874 generic.go:356] "Generic (PLEG): container finished" podID="d1950c75-bacc-4f16-91f0-66e2f6b3cf18" containerID="cea7bc32848dcb9f0ee5de150302cf6407d373338ff7661747152d052bba5c21" exitCode=0
Oct 30 04:29:58.041457 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.041307    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2" event={"ID":"d1950c75-bacc-4f16-91f0-66e2f6b3cf18","Type":"ContainerDied","Data":"cea7bc32848dcb9f0ee5de150302cf6407d373338ff7661747152d052bba5c21"}
Oct 30 04:29:58.076825 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.076798    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9prc6\" (UniqueName: \"kubernetes.io/projected/1d4d2573-179f-46bc-be32-e3dcc140437f-kube-api-access-9prc6\") pod \"payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n\" (UID: \"1d4d2573-179f-46bc-be32-e3dcc140437f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n"
Oct 30 04:29:58.149629 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.149602    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2"
Oct 30 04:29:58.200869 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.200412    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n"
Oct 30 04:29:58.237810 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.237771    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-openshift-service-ca\") pod \"d1950c75-bacc-4f16-91f0-66e2f6b3cf18\" (UID: \"d1950c75-bacc-4f16-91f0-66e2f6b3cf18\") "
Oct 30 04:29:58.237962 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.237833    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-kube-api-access-v5ml7\" (UniqueName: \"kubernetes.io/projected/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-kube-api-access-v5ml7\") pod \"d1950c75-bacc-4f16-91f0-66e2f6b3cf18\" (UID: \"d1950c75-bacc-4f16-91f0-66e2f6b3cf18\") "
Oct 30 04:29:58.237962 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.237932    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-metrics-certs\") pod \"d1950c75-bacc-4f16-91f0-66e2f6b3cf18\" (UID: \"d1950c75-bacc-4f16-91f0-66e2f6b3cf18\") "
Oct 30 04:29:58.238131 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.238044    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-maas-api-tls\") pod \"d1950c75-bacc-4f16-91f0-66e2f6b3cf18\" (UID: \"d1950c75-bacc-4f16-91f0-66e2f6b3cf18\") "
Oct 30 04:29:58.238836 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.238785    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-openshift-service-ca" pod "d1950c75-bacc-4f16-91f0-66e2f6b3cf18" (UID: "d1950c75-bacc-4f16-91f0-66e2f6b3cf18"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:29:58.240758 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.240733    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-kube-api-access-v5ml7" pod "d1950c75-bacc-4f16-91f0-66e2f6b3cf18" (UID: "d1950c75-bacc-4f16-91f0-66e2f6b3cf18"). InnerVolumeSpecName "kube-api-access-v5ml7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:29:58.241425 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.241387    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-maas-api-tls" pod "d1950c75-bacc-4f16-91f0-66e2f6b3cf18" (UID: "d1950c75-bacc-4f16-91f0-66e2f6b3cf18"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:29:58.241863 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.241838    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-metrics-certs" pod "d1950c75-bacc-4f16-91f0-66e2f6b3cf18" (UID: "d1950c75-bacc-4f16-91f0-66e2f6b3cf18"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:29:58.309048 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.308020    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk"
Oct 30 04:29:58.314104 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.314039    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz"
Oct 30 04:29:58.340586 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.340454    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:58.340586 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.340488    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5ml7\" (UniqueName: \"kubernetes.io/projected/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-kube-api-access-v5ml7\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:58.340586 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.340503    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:58.340586 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.340517    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1950c75-bacc-4f16-91f0-66e2f6b3cf18-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:58.551264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.464193    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d128d0ee-3fa4-4023-9333-cb7fa4ff5940-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d128d0ee-3fa4-4023-9333-cb7fa4ff5940-plugins-config-volume\") pod \"d128d0ee-3fa4-4023-9333-cb7fa4ff5940\" (UID: \"d128d0ee-3fa4-4023-9333-cb7fa4ff5940\") "
Oct 30 04:29:58.551264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.464288    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54704513-6246-41c9-978f-77b8d99e085b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54704513-6246-41c9-978f-77b8d99e085b-plugins-config-volume\") pod \"54704513-6246-41c9-978f-77b8d99e085b\" (UID: \"54704513-6246-41c9-978f-77b8d99e085b\") "
Oct 30 04:29:58.551264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.464370    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d128d0ee-3fa4-4023-9333-cb7fa4ff5940-kube-api-access-7t4sd\" (UniqueName: \"kubernetes.io/projected/d128d0ee-3fa4-4023-9333-cb7fa4ff5940-kube-api-access-7t4sd\") pod \"d128d0ee-3fa4-4023-9333-cb7fa4ff5940\" (UID: \"d128d0ee-3fa4-4023-9333-cb7fa4ff5940\") "
Oct 30 04:29:58.551264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.464420    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54704513-6246-41c9-978f-77b8d99e085b-kube-api-access-vzbq6\" (UniqueName: \"kubernetes.io/projected/54704513-6246-41c9-978f-77b8d99e085b-kube-api-access-vzbq6\") pod \"54704513-6246-41c9-978f-77b8d99e085b\" (UID: \"54704513-6246-41c9-978f-77b8d99e085b\") "
Oct 30 04:29:58.551264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.487115    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d128d0ee-3fa4-4023-9333-cb7fa4ff5940-plugins-config-volume" pod "d128d0ee-3fa4-4023-9333-cb7fa4ff5940" (UID: "d128d0ee-3fa4-4023-9333-cb7fa4ff5940"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:29:58.551264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.513524    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54704513-6246-41c9-978f-77b8d99e085b-plugins-config-volume" pod "54704513-6246-41c9-978f-77b8d99e085b" (UID: "54704513-6246-41c9-978f-77b8d99e085b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:29:58.562356 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.559888    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d128d0ee-3fa4-4023-9333-cb7fa4ff5940-kube-api-access-7t4sd" pod "d128d0ee-3fa4-4023-9333-cb7fa4ff5940" (UID: "d128d0ee-3fa4-4023-9333-cb7fa4ff5940"). InnerVolumeSpecName "kube-api-access-7t4sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:29:58.562356 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.562230    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54704513-6246-41c9-978f-77b8d99e085b-kube-api-access-vzbq6" pod "54704513-6246-41c9-978f-77b8d99e085b" (UID: "54704513-6246-41c9-978f-77b8d99e085b"). InnerVolumeSpecName "kube-api-access-vzbq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:29:58.577011 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.576974    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n"]
Oct 30 04:29:58.582151 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:29:58.582124    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d4d2573_179f_46bc_be32_e3dcc140437f.slice/crio-a235dd90cbceee370b8038f5b7b2111cbcd622068779f4c5d7c8886badc34082 WatchSource:0}: Error finding container a235dd90cbceee370b8038f5b7b2111cbcd622068779f4c5d7c8886badc34082: Status 404 returned error can't find the container with id a235dd90cbceee370b8038f5b7b2111cbcd622068779f4c5d7c8886badc34082
Oct 30 04:29:58.586844 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.586823    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:29:58.651347 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.651313    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-credential-socket\") pod \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") "
Oct 30 04:29:58.651464 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.651452    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba8e0c94-8f6b-40dd-86f9-3973177dd696-kube-api-access-5ktnk\" (UniqueName: \"kubernetes.io/projected/ba8e0c94-8f6b-40dd-86f9-3973177dd696-kube-api-access-5ktnk\") pod \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") "
Oct 30 04:29:58.651577 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.651475    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istiod-ca-cert\") pod \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") "
Oct 30 04:29:58.651577 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.651499    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-workload-socket\") pod \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") "
Oct 30 04:29:58.651577 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.651546    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-envoy\") pod \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") "
Oct 30 04:29:58.651774 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.651587    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-token\" (UniqueName: \"kubernetes.io/projected/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-token\") pod \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") "
Oct 30 04:29:58.651774 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.651604    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-credential-socket" pod "ba8e0c94-8f6b-40dd-86f9-3973177dd696" (UID: "ba8e0c94-8f6b-40dd-86f9-3973177dd696"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:29:58.651774 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.651620    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-podinfo\") pod \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") "
Oct 30 04:29:58.651774 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.651651    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-data\") pod \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") "
Oct 30 04:29:58.651774 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.651682    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-workload-certs\") pod \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\" (UID: \"ba8e0c94-8f6b-40dd-86f9-3973177dd696\") "
Oct 30 04:29:58.652025 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.651864    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istiod-ca-cert" pod "ba8e0c94-8f6b-40dd-86f9-3973177dd696" (UID: "ba8e0c94-8f6b-40dd-86f9-3973177dd696"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:29:58.652100 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.652073    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-workload-certs" pod "ba8e0c94-8f6b-40dd-86f9-3973177dd696" (UID: "ba8e0c94-8f6b-40dd-86f9-3973177dd696"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:29:58.652355 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.652329    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:58.652474 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.652358    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d128d0ee-3fa4-4023-9333-cb7fa4ff5940-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:58.652474 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.652372    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54704513-6246-41c9-978f-77b8d99e085b-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:58.652474 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.652386    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:58.652474 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.652403    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t4sd\" (UniqueName: \"kubernetes.io/projected/d128d0ee-3fa4-4023-9333-cb7fa4ff5940-kube-api-access-7t4sd\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:58.652474 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.652418    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzbq6\" (UniqueName: \"kubernetes.io/projected/54704513-6246-41c9-978f-77b8d99e085b-kube-api-access-vzbq6\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:58.652474 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.652422    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-data" pod "ba8e0c94-8f6b-40dd-86f9-3973177dd696" (UID: "ba8e0c94-8f6b-40dd-86f9-3973177dd696"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:29:58.652474 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.652431    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:58.652832 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.652626    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-workload-socket" pod "ba8e0c94-8f6b-40dd-86f9-3973177dd696" (UID: "ba8e0c94-8f6b-40dd-86f9-3973177dd696"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:29:58.654063 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.654035    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-envoy" pod "ba8e0c94-8f6b-40dd-86f9-3973177dd696" (UID: "ba8e0c94-8f6b-40dd-86f9-3973177dd696"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:29:58.654302 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.654282    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba8e0c94-8f6b-40dd-86f9-3973177dd696-kube-api-access-5ktnk" pod "ba8e0c94-8f6b-40dd-86f9-3973177dd696" (UID: "ba8e0c94-8f6b-40dd-86f9-3973177dd696"). InnerVolumeSpecName "kube-api-access-5ktnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:29:58.654763 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.654746    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-podinfo" pod "ba8e0c94-8f6b-40dd-86f9-3973177dd696" (UID: "ba8e0c94-8f6b-40dd-86f9-3973177dd696"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:29:58.654763 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.654749    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-token" pod "ba8e0c94-8f6b-40dd-86f9-3973177dd696" (UID: "ba8e0c94-8f6b-40dd-86f9-3973177dd696"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:29:58.753645 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.753622    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ktnk\" (UniqueName: \"kubernetes.io/projected/ba8e0c94-8f6b-40dd-86f9-3973177dd696-kube-api-access-5ktnk\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:58.753645 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.753646    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:58.753776 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.753657    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:58.753776 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.753666    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:58.753776 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.753675    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:58.753776 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:58.753682    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba8e0c94-8f6b-40dd-86f9-3973177dd696-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:29:59.048997 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.048964    4874 generic.go:356] "Generic (PLEG): container finished" podID="d128d0ee-3fa4-4023-9333-cb7fa4ff5940" containerID="16cc713a89d924e1831ee15a27825a562ada950d42b8ded553dc5bc50e141c94" exitCode=0
Oct 30 04:29:59.049480 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.049024    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk"
Oct 30 04:29:59.049480 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.049058    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk" event={"ID":"d128d0ee-3fa4-4023-9333-cb7fa4ff5940","Type":"ContainerDied","Data":"16cc713a89d924e1831ee15a27825a562ada950d42b8ded553dc5bc50e141c94"}
Oct 30 04:29:59.049480 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.049110    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk" event={"ID":"d128d0ee-3fa4-4023-9333-cb7fa4ff5940","Type":"ContainerDied","Data":"cd82df6b74e64a7b4270a5e16e745cf515e1422dad6e1202fab6768fed4b6cab"}
Oct 30 04:29:59.049480 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.049136    4874 scope.go:122] "RemoveContainer" containerID="16cc713a89d924e1831ee15a27825a562ada950d42b8ded553dc5bc50e141c94"
Oct 30 04:29:59.051049 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.051026    4874 generic.go:356] "Generic (PLEG): container finished" podID="ba8e0c94-8f6b-40dd-86f9-3973177dd696" containerID="ab03c3db0059bcc14c41937f07627fce74e16ef0ee6ad0a9df95c583d0405cff" exitCode=0
Oct 30 04:29:59.051257 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.051082    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"
Oct 30 04:29:59.051483 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.051086    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz" event={"ID":"ba8e0c94-8f6b-40dd-86f9-3973177dd696","Type":"ContainerDied","Data":"ab03c3db0059bcc14c41937f07627fce74e16ef0ee6ad0a9df95c583d0405cff"}
Oct 30 04:29:59.051578 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.051535    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz" event={"ID":"ba8e0c94-8f6b-40dd-86f9-3973177dd696","Type":"ContainerDied","Data":"a379274e28679c2a438590e7a6aab93bc80ef5721cdee61d37d23bb543ae4154"}
Oct 30 04:29:59.053190 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.053015    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n" event={"ID":"1d4d2573-179f-46bc-be32-e3dcc140437f","Type":"ContainerStarted","Data":"b9a4be4ff309eb77a20012241822eb52a263c5190148c877ae6324fa3dcd9a92"}
Oct 30 04:29:59.053190 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.053061    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n" event={"ID":"1d4d2573-179f-46bc-be32-e3dcc140437f","Type":"ContainerStarted","Data":"a235dd90cbceee370b8038f5b7b2111cbcd622068779f4c5d7c8886badc34082"}
Oct 30 04:29:59.053190 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.053123    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n"
Oct 30 04:29:59.054523 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.054498    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2" event={"ID":"d1950c75-bacc-4f16-91f0-66e2f6b3cf18","Type":"ContainerDied","Data":"63d849418b929f375e0be3433c16961a4f84f6626419a6427ab225a70dc29af3"}
Oct 30 04:29:59.054629 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.054514    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2"
Oct 30 04:29:59.055832 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.055810    4874 generic.go:356] "Generic (PLEG): container finished" podID="54704513-6246-41c9-978f-77b8d99e085b" containerID="de1a7fed98e7620a237d9b0f21072f7c2fa3f76afbdb4366a3a37c1e81449a19" exitCode=0
Oct 30 04:29:59.055913 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.055860    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz"
Oct 30 04:29:59.055913 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.055896    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz" event={"ID":"54704513-6246-41c9-978f-77b8d99e085b","Type":"ContainerDied","Data":"de1a7fed98e7620a237d9b0f21072f7c2fa3f76afbdb4366a3a37c1e81449a19"}
Oct 30 04:29:59.056033 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.055941    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz" event={"ID":"54704513-6246-41c9-978f-77b8d99e085b","Type":"ContainerDied","Data":"cc2cacb78a10b0c79308aed9648a0514de8de8cb070d78b69abad2e1a46703f7"}
Oct 30 04:29:59.059613 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.059595    4874 scope.go:122] "RemoveContainer" containerID="16cc713a89d924e1831ee15a27825a562ada950d42b8ded553dc5bc50e141c94"
Oct 30 04:29:59.059865 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:29:59.059844    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16cc713a89d924e1831ee15a27825a562ada950d42b8ded553dc5bc50e141c94\": container with ID starting with 16cc713a89d924e1831ee15a27825a562ada950d42b8ded553dc5bc50e141c94 not found: ID does not exist" containerID="16cc713a89d924e1831ee15a27825a562ada950d42b8ded553dc5bc50e141c94"
Oct 30 04:29:59.059957 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.059871    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16cc713a89d924e1831ee15a27825a562ada950d42b8ded553dc5bc50e141c94"} err="failed to get container status \"16cc713a89d924e1831ee15a27825a562ada950d42b8ded553dc5bc50e141c94\": rpc error: code = NotFound desc = could not find container \"16cc713a89d924e1831ee15a27825a562ada950d42b8ded553dc5bc50e141c94\": container with ID starting with 16cc713a89d924e1831ee15a27825a562ada950d42b8ded553dc5bc50e141c94 not found: ID does not exist"
Oct 30 04:29:59.059957 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.059887    4874 scope.go:122] "RemoveContainer" containerID="ab03c3db0059bcc14c41937f07627fce74e16ef0ee6ad0a9df95c583d0405cff"
Oct 30 04:29:59.068565 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.068548    4874 scope.go:122] "RemoveContainer" containerID="ed12f2000ad6f93266022a3c200402a5513443b3ed59dde9737d68edfe9eff4e"
Oct 30 04:29:59.076755 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.076728    4874 scope.go:122] "RemoveContainer" containerID="ab03c3db0059bcc14c41937f07627fce74e16ef0ee6ad0a9df95c583d0405cff"
Oct 30 04:29:59.077009 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:29:59.076989    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab03c3db0059bcc14c41937f07627fce74e16ef0ee6ad0a9df95c583d0405cff\": container with ID starting with ab03c3db0059bcc14c41937f07627fce74e16ef0ee6ad0a9df95c583d0405cff not found: ID does not exist" containerID="ab03c3db0059bcc14c41937f07627fce74e16ef0ee6ad0a9df95c583d0405cff"
Oct 30 04:29:59.077082 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.077013    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab03c3db0059bcc14c41937f07627fce74e16ef0ee6ad0a9df95c583d0405cff"} err="failed to get container status \"ab03c3db0059bcc14c41937f07627fce74e16ef0ee6ad0a9df95c583d0405cff\": rpc error: code = NotFound desc = could not find container \"ab03c3db0059bcc14c41937f07627fce74e16ef0ee6ad0a9df95c583d0405cff\": container with ID starting with ab03c3db0059bcc14c41937f07627fce74e16ef0ee6ad0a9df95c583d0405cff not found: ID does not exist"
Oct 30 04:29:59.077082 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.077030    4874 scope.go:122] "RemoveContainer" containerID="ed12f2000ad6f93266022a3c200402a5513443b3ed59dde9737d68edfe9eff4e"
Oct 30 04:29:59.077329 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:29:59.077306    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed12f2000ad6f93266022a3c200402a5513443b3ed59dde9737d68edfe9eff4e\": container with ID starting with ed12f2000ad6f93266022a3c200402a5513443b3ed59dde9737d68edfe9eff4e not found: ID does not exist" containerID="ed12f2000ad6f93266022a3c200402a5513443b3ed59dde9737d68edfe9eff4e"
Oct 30 04:29:59.077373 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.077337    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed12f2000ad6f93266022a3c200402a5513443b3ed59dde9737d68edfe9eff4e"} err="failed to get container status \"ed12f2000ad6f93266022a3c200402a5513443b3ed59dde9737d68edfe9eff4e\": rpc error: code = NotFound desc = could not find container \"ed12f2000ad6f93266022a3c200402a5513443b3ed59dde9737d68edfe9eff4e\": container with ID starting with ed12f2000ad6f93266022a3c200402a5513443b3ed59dde9737d68edfe9eff4e not found: ID does not exist"
Oct 30 04:29:59.077373 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.077364    4874 scope.go:122] "RemoveContainer" containerID="cea7bc32848dcb9f0ee5de150302cf6407d373338ff7661747152d052bba5c21"
Oct 30 04:29:59.087610 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.087588    4874 scope.go:122] "RemoveContainer" containerID="de1a7fed98e7620a237d9b0f21072f7c2fa3f76afbdb4366a3a37c1e81449a19"
Oct 30 04:29:59.095376 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.095360    4874 scope.go:122] "RemoveContainer" containerID="de1a7fed98e7620a237d9b0f21072f7c2fa3f76afbdb4366a3a37c1e81449a19"
Oct 30 04:29:59.095635 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:29:59.095614    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de1a7fed98e7620a237d9b0f21072f7c2fa3f76afbdb4366a3a37c1e81449a19\": container with ID starting with de1a7fed98e7620a237d9b0f21072f7c2fa3f76afbdb4366a3a37c1e81449a19 not found: ID does not exist" containerID="de1a7fed98e7620a237d9b0f21072f7c2fa3f76afbdb4366a3a37c1e81449a19"
Oct 30 04:29:59.095697 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.095645    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de1a7fed98e7620a237d9b0f21072f7c2fa3f76afbdb4366a3a37c1e81449a19"} err="failed to get container status \"de1a7fed98e7620a237d9b0f21072f7c2fa3f76afbdb4366a3a37c1e81449a19\": rpc error: code = NotFound desc = could not find container \"de1a7fed98e7620a237d9b0f21072f7c2fa3f76afbdb4366a3a37c1e81449a19\": container with ID starting with de1a7fed98e7620a237d9b0f21072f7c2fa3f76afbdb4366a3a37c1e81449a19 not found: ID does not exist"
Oct 30 04:29:59.098773 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.098748    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"]
Oct 30 04:29:59.109920 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.109873    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b1a417-openshift-default-647576dbdf-tvlxz"]
Oct 30 04:29:59.135679 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.135653    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz"]
Oct 30 04:29:59.137811 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.137794    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-763bde-77f9dc9ff9-gkwvz"]
Oct 30 04:29:59.169508 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.169486    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2"]
Oct 30 04:29:59.173284 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.173263    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-763bde-74bcd9967d-njjf2"]
Oct 30 04:29:59.215371 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.215347    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk"]
Oct 30 04:29:59.217277 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.217255    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-763bde-d79cb8b7d-d7ttk"]
Oct 30 04:29:59.256411 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:29:59.256365    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n" podStartSLOduration=2.256354906 podStartE2EDuration="2.256354906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:29:59.25466365 +0000 UTC m=+1918.667059224" watchObservedRunningTime="2026-10-30 04:29:59.256354906 +0000 UTC m=+1918.668750478"
Oct 30 04:30:00.729751 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:00.729719    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54704513-6246-41c9-978f-77b8d99e085b" path="/var/lib/kubelet/pods/54704513-6246-41c9-978f-77b8d99e085b/volumes"
Oct 30 04:30:00.730143 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:00.730129    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba8e0c94-8f6b-40dd-86f9-3973177dd696" path="/var/lib/kubelet/pods/ba8e0c94-8f6b-40dd-86f9-3973177dd696/volumes"
Oct 30 04:30:00.730525 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:00.730512    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d128d0ee-3fa4-4023-9333-cb7fa4ff5940" path="/var/lib/kubelet/pods/d128d0ee-3fa4-4023-9333-cb7fa4ff5940/volumes"
Oct 30 04:30:00.731158 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:00.731142    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1950c75-bacc-4f16-91f0-66e2f6b3cf18" path="/var/lib/kubelet/pods/d1950c75-bacc-4f16-91f0-66e2f6b3cf18/volumes"
Oct 30 04:30:01.206060 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.206030    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr"]
Oct 30 04:30:01.206574 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.206557    4874 cpu_manager.go:414] "containerMap: removing container" podUID="ba8e0c94-8f6b-40dd-86f9-3973177dd696" containerName="istio-proxy"
Oct 30 04:30:01.206668 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.206576    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba8e0c94-8f6b-40dd-86f9-3973177dd696" containerName="istio-proxy"
Oct 30 04:30:01.206668 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.206605    4874 cpu_manager.go:414] "containerMap: removing container" podUID="d128d0ee-3fa4-4023-9333-cb7fa4ff5940" containerName="payload-processing"
Oct 30 04:30:01.206668 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.206613    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d128d0ee-3fa4-4023-9333-cb7fa4ff5940" containerName="payload-processing"
Oct 30 04:30:01.206668 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.206636    4874 cpu_manager.go:414] "containerMap: removing container" podUID="d1950c75-bacc-4f16-91f0-66e2f6b3cf18" containerName="maas-api"
Oct 30 04:30:01.206668 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.206643    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1950c75-bacc-4f16-91f0-66e2f6b3cf18" containerName="maas-api"
Oct 30 04:30:01.206668 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.206665    4874 cpu_manager.go:414] "containerMap: removing container" podUID="54704513-6246-41c9-978f-77b8d99e085b" containerName="payload-pre-processing"
Oct 30 04:30:01.206953 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.206673    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54704513-6246-41c9-978f-77b8d99e085b" containerName="payload-pre-processing"
Oct 30 04:30:01.206953 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.206804    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba8e0c94-8f6b-40dd-86f9-3973177dd696" containerName="istio-proxy"
Oct 30 04:30:01.206953 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.206819    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="d128d0ee-3fa4-4023-9333-cb7fa4ff5940" containerName="payload-processing"
Oct 30 04:30:01.206953 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.206831    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1950c75-bacc-4f16-91f0-66e2f6b3cf18" containerName="maas-api"
Oct 30 04:30:01.206953 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.206843    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="54704513-6246-41c9-978f-77b8d99e085b" containerName="payload-pre-processing"
Oct 30 04:30:01.212363 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.212344    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr"
Oct 30 04:30:01.229312 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.229288    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr"]
Oct 30 04:30:01.237923 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.237899    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-71b161-dockercfg-45k69\""
Oct 30 04:30:01.237923 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.237901    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-71b161\""
Oct 30 04:30:01.282530 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.282498    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b394ae-aff0-45fb-947e-05cd7bbd436a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr\" (UID: \"71b394ae-aff0-45fb-947e-05cd7bbd436a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr"
Oct 30 04:30:01.282726 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.282701    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76rdv\" (UniqueName: \"kubernetes.io/projected/71b394ae-aff0-45fb-947e-05cd7bbd436a-kube-api-access-76rdv\") pod \"payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr\" (UID: \"71b394ae-aff0-45fb-947e-05cd7bbd436a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr"
Oct 30 04:30:01.370261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.364557    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-71b161-668476b448-8489m"]
Oct 30 04:30:01.370261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.365348    4874 cpu_manager.go:414] "containerMap: removing container" podUID="ba8e0c94-8f6b-40dd-86f9-3973177dd696" containerName="istio-proxy"
Oct 30 04:30:01.370261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.365368    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba8e0c94-8f6b-40dd-86f9-3973177dd696" containerName="istio-proxy"
Oct 30 04:30:01.370261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.365531    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba8e0c94-8f6b-40dd-86f9-3973177dd696" containerName="istio-proxy"
Oct 30 04:30:01.391312 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.389220    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76rdv\" (UniqueName: \"kubernetes.io/projected/71b394ae-aff0-45fb-947e-05cd7bbd436a-kube-api-access-76rdv\") pod \"payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr\" (UID: \"71b394ae-aff0-45fb-947e-05cd7bbd436a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr"
Oct 30 04:30:01.391312 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.389385    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b394ae-aff0-45fb-947e-05cd7bbd436a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr\" (UID: \"71b394ae-aff0-45fb-947e-05cd7bbd436a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr"
Oct 30 04:30:01.391312 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.390693    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b394ae-aff0-45fb-947e-05cd7bbd436a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr\" (UID: \"71b394ae-aff0-45fb-947e-05cd7bbd436a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr"
Oct 30 04:30:01.395260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.393421    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-71b161-668476b448-8489m"]
Oct 30 04:30:01.395260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.393534    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-71b161-668476b448-8489m"
Oct 30 04:30:01.442714 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.439280    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76rdv\" (UniqueName: \"kubernetes.io/projected/71b394ae-aff0-45fb-947e-05cd7bbd436a-kube-api-access-76rdv\") pod \"payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr\" (UID: \"71b394ae-aff0-45fb-947e-05cd7bbd436a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr"
Oct 30 04:30:01.490436 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.490405    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq2sq\" (UniqueName: \"kubernetes.io/projected/96a65bb0-e452-4068-ba0e-7145782c3b1f-kube-api-access-qq2sq\") pod \"payload-processing-e2e-praxis-a-71b161-668476b448-8489m\" (UID: \"96a65bb0-e452-4068-ba0e-7145782c3b1f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-71b161-668476b448-8489m"
Oct 30 04:30:01.490615 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.490572    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a65bb0-e452-4068-ba0e-7145782c3b1f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-71b161-668476b448-8489m\" (UID: \"96a65bb0-e452-4068-ba0e-7145782c3b1f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-71b161-668476b448-8489m"
Oct 30 04:30:01.525007 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.524980    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr"
Oct 30 04:30:01.609123 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.596492    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a65bb0-e452-4068-ba0e-7145782c3b1f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-71b161-668476b448-8489m\" (UID: \"96a65bb0-e452-4068-ba0e-7145782c3b1f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-71b161-668476b448-8489m"
Oct 30 04:30:01.609123 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.596640    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq2sq\" (UniqueName: \"kubernetes.io/projected/96a65bb0-e452-4068-ba0e-7145782c3b1f-kube-api-access-qq2sq\") pod \"payload-processing-e2e-praxis-a-71b161-668476b448-8489m\" (UID: \"96a65bb0-e452-4068-ba0e-7145782c3b1f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-71b161-668476b448-8489m"
Oct 30 04:30:01.609123 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.597779    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a65bb0-e452-4068-ba0e-7145782c3b1f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-71b161-668476b448-8489m\" (UID: \"96a65bb0-e452-4068-ba0e-7145782c3b1f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-71b161-668476b448-8489m"
Oct 30 04:30:01.621427 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.621397    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq2sq\" (UniqueName: \"kubernetes.io/projected/96a65bb0-e452-4068-ba0e-7145782c3b1f-kube-api-access-qq2sq\") pod \"payload-processing-e2e-praxis-a-71b161-668476b448-8489m\" (UID: \"96a65bb0-e452-4068-ba0e-7145782c3b1f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-71b161-668476b448-8489m"
Oct 30 04:30:01.734683 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.734658    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-71b161-668476b448-8489m"
Oct 30 04:30:01.827614 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:01.827586    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr"]
Oct 30 04:30:01.830001 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:30:01.829974    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b394ae_aff0_45fb_947e_05cd7bbd436a.slice/crio-c6c18a8f16159797d29e57f747912276960060a44bc9a9bedc125c628b73bdae WatchSource:0}: Error finding container c6c18a8f16159797d29e57f747912276960060a44bc9a9bedc125c628b73bdae: Status 404 returned error can't find the container with id c6c18a8f16159797d29e57f747912276960060a44bc9a9bedc125c628b73bdae
Oct 30 04:30:02.047114 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:02.047091    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-71b161-668476b448-8489m"]
Oct 30 04:30:02.049575 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:30:02.049545    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a65bb0_e452_4068_ba0e_7145782c3b1f.slice/crio-685c302c1408926e017ae0bb96c57bf8456cd2bfaa57836ac6ca6458a4fff0e3 WatchSource:0}: Error finding container 685c302c1408926e017ae0bb96c57bf8456cd2bfaa57836ac6ca6458a4fff0e3: Status 404 returned error can't find the container with id 685c302c1408926e017ae0bb96c57bf8456cd2bfaa57836ac6ca6458a4fff0e3
Oct 30 04:30:02.113722 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:02.113692    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-71b161-668476b448-8489m" event={"ID":"96a65bb0-e452-4068-ba0e-7145782c3b1f","Type":"ContainerStarted","Data":"685c302c1408926e017ae0bb96c57bf8456cd2bfaa57836ac6ca6458a4fff0e3"}
Oct 30 04:30:02.115307 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:02.115283    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr" event={"ID":"71b394ae-aff0-45fb-947e-05cd7bbd436a","Type":"ContainerStarted","Data":"ff8d8d0d40bba0e262adb0d9fb64f1b902805332711d8162aef8733d591d7cb9"}
Oct 30 04:30:02.115401 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:02.115329    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr" event={"ID":"71b394ae-aff0-45fb-947e-05cd7bbd436a","Type":"ContainerStarted","Data":"c6c18a8f16159797d29e57f747912276960060a44bc9a9bedc125c628b73bdae"}
Oct 30 04:30:02.115401 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:02.115390    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr"
Oct 30 04:30:02.156601 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:02.156553    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr" podStartSLOduration=1.156538783 podStartE2EDuration="1.156538783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:02.154381663 +0000 UTC m=+1921.566777236" watchObservedRunningTime="2026-10-30 04:30:02.156538783 +0000 UTC m=+1921.568934356"
Oct 30 04:30:03.129926 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:03.129869    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-71b161-668476b448-8489m" event={"ID":"96a65bb0-e452-4068-ba0e-7145782c3b1f","Type":"ContainerStarted","Data":"323935c9e1f07702be8b088c545084688c58b4c4830f737504790294c5b0fe7b"}
Oct 30 04:30:03.130574 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:03.130538    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-71b161-668476b448-8489m"
Oct 30 04:30:03.171413 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:03.171356    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-71b161-668476b448-8489m" podStartSLOduration=2.171339388 podStartE2EDuration="2.171339388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:03.169186616 +0000 UTC m=+1922.581582258" watchObservedRunningTime="2026-10-30 04:30:03.171339388 +0000 UTC m=+1922.583734962"
Oct 30 04:30:03.821519 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:03.821486    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:30:06.332889 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:06.332396    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj"]
Oct 30 04:30:06.350261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:06.349877    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj"
Oct 30 04:30:06.373015 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:06.372983    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3f5508\""
Oct 30 04:30:06.373154 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:06.373100    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3f5508-dockercfg-5lq87\""
Oct 30 04:30:06.376867 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:06.376839    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj"]
Oct 30 04:30:06.478415 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:06.478377    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njqdh\" (UniqueName: \"kubernetes.io/projected/0622771d-893d-40bf-8397-f3b06a4e62f1-kube-api-access-njqdh\") pod \"payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj\" (UID: \"0622771d-893d-40bf-8397-f3b06a4e62f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj"
Oct 30 04:30:06.478568 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:06.478506    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0622771d-893d-40bf-8397-f3b06a4e62f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj\" (UID: \"0622771d-893d-40bf-8397-f3b06a4e62f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj"
Oct 30 04:30:06.480910 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:06.480886    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6"]
Oct 30 04:30:06.484505 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:06.484490    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6"
Oct 30 04:30:06.505813 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:06.505792    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6"]
Oct 30 04:30:06.579752 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:06.579725    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njqdh\" (UniqueName: \"kubernetes.io/projected/0622771d-893d-40bf-8397-f3b06a4e62f1-kube-api-access-njqdh\") pod \"payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj\" (UID: \"0622771d-893d-40bf-8397-f3b06a4e62f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj"
Oct 30 04:30:06.579867 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:06.579776    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0622771d-893d-40bf-8397-f3b06a4e62f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj\" (UID: \"0622771d-893d-40bf-8397-f3b06a4e62f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj"
Oct 30 04:30:06.579867 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:06.579814    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxrpq\" (UniqueName: \"kubernetes.io/projected/6b9e2a70-701b-42c7-967c-70101a99e94e-kube-api-access-xxrpq\") pod \"payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6\" (UID: \"6b9e2a70-701b-42c7-967c-70101a99e94e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6"
Oct 30 04:30:06.579867 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:06.579859    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b9e2a70-701b-42c7-967c-70101a99e94e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6\" (UID: \"6b9e2a70-701b-42c7-967c-70101a99e94e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6"
Oct 30 04:30:06.580497 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:06.580478    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0622771d-893d-40bf-8397-f3b06a4e62f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj\" (UID: \"0622771d-893d-40bf-8397-f3b06a4e62f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj"
Oct 30 04:30:06.618916 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:06.618865    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njqdh\" (UniqueName: \"kubernetes.io/projected/0622771d-893d-40bf-8397-f3b06a4e62f1-kube-api-access-njqdh\") pod \"payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj\" (UID: \"0622771d-893d-40bf-8397-f3b06a4e62f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj"
Oct 30 04:30:06.681468 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:06.681446    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxrpq\" (UniqueName: \"kubernetes.io/projected/6b9e2a70-701b-42c7-967c-70101a99e94e-kube-api-access-xxrpq\") pod \"payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6\" (UID: \"6b9e2a70-701b-42c7-967c-70101a99e94e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6"
Oct 30 04:30:06.681561 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:06.681489    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b9e2a70-701b-42c7-967c-70101a99e94e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6\" (UID: \"6b9e2a70-701b-42c7-967c-70101a99e94e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6"
Oct 30 04:30:06.682090 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:06.682072    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b9e2a70-701b-42c7-967c-70101a99e94e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6\" (UID: \"6b9e2a70-701b-42c7-967c-70101a99e94e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6"
Oct 30 04:30:06.691094 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:06.691076    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj"
Oct 30 04:30:06.734269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:06.732038    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxrpq\" (UniqueName: \"kubernetes.io/projected/6b9e2a70-701b-42c7-967c-70101a99e94e-kube-api-access-xxrpq\") pod \"payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6\" (UID: \"6b9e2a70-701b-42c7-967c-70101a99e94e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6"
Oct 30 04:30:06.798061 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:06.798034    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6"
Oct 30 04:30:06.998607 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:06.998578    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj"]
Oct 30 04:30:07.001972 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:30:07.001944    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0622771d_893d_40bf_8397_f3b06a4e62f1.slice/crio-3ffefc8721823f99ede7962f027cc4862c01eab721e09a872381680fcf4eda24 WatchSource:0}: Error finding container 3ffefc8721823f99ede7962f027cc4862c01eab721e09a872381680fcf4eda24: Status 404 returned error can't find the container with id 3ffefc8721823f99ede7962f027cc4862c01eab721e09a872381680fcf4eda24
Oct 30 04:30:07.101511 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:07.101424    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6"]
Oct 30 04:30:07.103889 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:30:07.103864    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b9e2a70_701b_42c7_967c_70101a99e94e.slice/crio-506ce4190762eb9840d71edb73d7805dc2f0d02d74febf0a62a6020dc40ac063 WatchSource:0}: Error finding container 506ce4190762eb9840d71edb73d7805dc2f0d02d74febf0a62a6020dc40ac063: Status 404 returned error can't find the container with id 506ce4190762eb9840d71edb73d7805dc2f0d02d74febf0a62a6020dc40ac063
Oct 30 04:30:07.191672 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:07.191408    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6" event={"ID":"6b9e2a70-701b-42c7-967c-70101a99e94e","Type":"ContainerStarted","Data":"506ce4190762eb9840d71edb73d7805dc2f0d02d74febf0a62a6020dc40ac063"}
Oct 30 04:30:07.199564 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:07.199527    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj" event={"ID":"0622771d-893d-40bf-8397-f3b06a4e62f1","Type":"ContainerStarted","Data":"243a0e13eed7fc7cb8446c7cadc00acb1893b71c8050cfe8c7e917002a706b0b"}
Oct 30 04:30:07.199716 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:07.199589    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj" event={"ID":"0622771d-893d-40bf-8397-f3b06a4e62f1","Type":"ContainerStarted","Data":"3ffefc8721823f99ede7962f027cc4862c01eab721e09a872381680fcf4eda24"}
Oct 30 04:30:07.199716 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:07.199655    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj"
Oct 30 04:30:07.239696 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:07.239625    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj" podStartSLOduration=1.239607111 podStartE2EDuration="1.239607111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:07.237881187 +0000 UTC m=+1926.650276773" watchObservedRunningTime="2026-10-30 04:30:07.239607111 +0000 UTC m=+1926.652002685"
Oct 30 04:30:07.326018 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:07.325944    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"]
Oct 30 04:30:07.326157 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:07.326133    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh" podUID="6a80cae8-0b58-4643-aadb-771c6ea2651a" containerName="istio-proxy" containerID="cri-o://85184c52773421c64459159dfaf6c0029b07dce151dca6d02948879f987788f4" gracePeriod=30
Oct 30 04:30:08.044002 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:08.043976    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9"
Oct 30 04:30:08.231329 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:08.231295    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6" event={"ID":"6b9e2a70-701b-42c7-967c-70101a99e94e","Type":"ContainerStarted","Data":"b431354ad001a66327061da7fd7ff9de70d10f7df9076cab8cfacd60a1603a98"}
Oct 30 04:30:08.231638 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:08.231617    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6"
Oct 30 04:30:08.273176 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:08.273128    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6" podStartSLOduration=2.273113614 podStartE2EDuration="2.273113614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:08.272836737 +0000 UTC m=+1927.685232307" watchObservedRunningTime="2026-10-30 04:30:08.273113614 +0000 UTC m=+1927.685509186"
Oct 30 04:30:10.103518 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:10.103490    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n"
Oct 30 04:30:10.731597 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:10.731567    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm"]
Oct 30 04:30:10.735727 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:10.735703    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm"
Oct 30 04:30:10.741227 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:10.741205    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a444bc-289c-49df-a4e0-e85a241e73dc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm\" (UID: \"d8a444bc-289c-49df-a4e0-e85a241e73dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm"
Oct 30 04:30:10.741340 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:10.741263    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a444bc-289c-49df-a4e0-e85a241e73dc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm\" (UID: \"d8a444bc-289c-49df-a4e0-e85a241e73dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm"
Oct 30 04:30:10.741392 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:10.741377    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8a444bc-289c-49df-a4e0-e85a241e73dc-metrics-certs\") pod \"maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm\" (UID: \"d8a444bc-289c-49df-a4e0-e85a241e73dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm"
Oct 30 04:30:10.741434 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:10.741405    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8s5m\" (UniqueName: \"kubernetes.io/projected/d8a444bc-289c-49df-a4e0-e85a241e73dc-kube-api-access-c8s5m\") pod \"maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm\" (UID: \"d8a444bc-289c-49df-a4e0-e85a241e73dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm"
Oct 30 04:30:10.765220 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:10.765198    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm"]
Oct 30 04:30:10.796728 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:10.796696    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-dbabed-serving-cert\""
Oct 30 04:30:10.841965 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:10.841932    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a444bc-289c-49df-a4e0-e85a241e73dc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm\" (UID: \"d8a444bc-289c-49df-a4e0-e85a241e73dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm"
Oct 30 04:30:10.842090 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:10.841992    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a444bc-289c-49df-a4e0-e85a241e73dc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm\" (UID: \"d8a444bc-289c-49df-a4e0-e85a241e73dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm"
Oct 30 04:30:10.842090 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:30:10.842067    4874 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-serving-cert: secret "maas-api-e2e-ipp-b-dbabed-serving-cert" not found
Oct 30 04:30:10.842201 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:30:10.842118    4874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8a444bc-289c-49df-a4e0-e85a241e73dc-maas-api-tls podName:d8a444bc-289c-49df-a4e0-e85a241e73dc nodeName:}" failed. No retries permitted until 2026-10-30 04:30:11.342104404 +0000 UTC m=+1930.754499958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d8a444bc-289c-49df-a4e0-e85a241e73dc-maas-api-tls") pod "maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm" (UID: "d8a444bc-289c-49df-a4e0-e85a241e73dc") : secret "maas-api-e2e-ipp-b-dbabed-serving-cert" not found
Oct 30 04:30:10.842201 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:10.842071    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8a444bc-289c-49df-a4e0-e85a241e73dc-metrics-certs\") pod \"maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm\" (UID: \"d8a444bc-289c-49df-a4e0-e85a241e73dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm"
Oct 30 04:30:10.842201 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:10.842195    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8s5m\" (UniqueName: \"kubernetes.io/projected/d8a444bc-289c-49df-a4e0-e85a241e73dc-kube-api-access-c8s5m\") pod \"maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm\" (UID: \"d8a444bc-289c-49df-a4e0-e85a241e73dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm"
Oct 30 04:30:10.842737 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:10.842719    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a444bc-289c-49df-a4e0-e85a241e73dc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm\" (UID: \"d8a444bc-289c-49df-a4e0-e85a241e73dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm"
Oct 30 04:30:10.844544 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:10.844523    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8a444bc-289c-49df-a4e0-e85a241e73dc-metrics-certs\") pod \"maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm\" (UID: \"d8a444bc-289c-49df-a4e0-e85a241e73dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm"
Oct 30 04:30:10.873285 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:10.873231    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8s5m\" (UniqueName: \"kubernetes.io/projected/d8a444bc-289c-49df-a4e0-e85a241e73dc-kube-api-access-c8s5m\") pod \"maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm\" (UID: \"d8a444bc-289c-49df-a4e0-e85a241e73dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm"
Oct 30 04:30:11.349617 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:11.349585    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a444bc-289c-49df-a4e0-e85a241e73dc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm\" (UID: \"d8a444bc-289c-49df-a4e0-e85a241e73dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm"
Oct 30 04:30:11.351937 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:11.351904    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a444bc-289c-49df-a4e0-e85a241e73dc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm\" (UID: \"d8a444bc-289c-49df-a4e0-e85a241e73dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm"
Oct 30 04:30:11.649617 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:11.649533    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm"
Oct 30 04:30:12.001369 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.001342    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm"]
Oct 30 04:30:12.003473 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:30:12.003440    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a444bc_289c_49df_a4e0_e85a241e73dc.slice/crio-f7bd0ad2c49e07d61e8b0e7acd886962c01e87b8a34d021b455872b0db5d37da WatchSource:0}: Error finding container f7bd0ad2c49e07d61e8b0e7acd886962c01e87b8a34d021b455872b0db5d37da: Status 404 returned error can't find the container with id f7bd0ad2c49e07d61e8b0e7acd886962c01e87b8a34d021b455872b0db5d37da
Oct 30 04:30:12.253251 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.253156    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm" event={"ID":"d8a444bc-289c-49df-a4e0-e85a241e73dc","Type":"ContainerStarted","Data":"f7bd0ad2c49e07d61e8b0e7acd886962c01e87b8a34d021b455872b0db5d37da"}
Oct 30 04:30:12.598890 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.598863    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:30:12.683541 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.683512    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a80cae8-0b58-4643-aadb-771c6ea2651a-kube-api-access-jj8p9\" (UniqueName: \"kubernetes.io/projected/6a80cae8-0b58-4643-aadb-771c6ea2651a-kube-api-access-jj8p9\") pod \"6a80cae8-0b58-4643-aadb-771c6ea2651a\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") "
Oct 30 04:30:12.683738 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.683600    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-envoy\") pod \"6a80cae8-0b58-4643-aadb-771c6ea2651a\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") "
Oct 30 04:30:12.683738 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.683640    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-data\") pod \"6a80cae8-0b58-4643-aadb-771c6ea2651a\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") "
Oct 30 04:30:12.683738 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.683713    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-token\" (UniqueName: \"kubernetes.io/projected/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-token\") pod \"6a80cae8-0b58-4643-aadb-771c6ea2651a\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") "
Oct 30 04:30:12.683888 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.683810    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-podinfo\") pod \"6a80cae8-0b58-4643-aadb-771c6ea2651a\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") "
Oct 30 04:30:12.683888 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.683856    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-credential-socket\") pod \"6a80cae8-0b58-4643-aadb-771c6ea2651a\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") "
Oct 30 04:30:12.683995 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.683902    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-workload-certs\") pod \"6a80cae8-0b58-4643-aadb-771c6ea2651a\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") "
Oct 30 04:30:12.683995 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.683942    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-workload-socket\") pod \"6a80cae8-0b58-4643-aadb-771c6ea2651a\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") "
Oct 30 04:30:12.684867 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.684351    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-workload-certs" pod "6a80cae8-0b58-4643-aadb-771c6ea2651a" (UID: "6a80cae8-0b58-4643-aadb-771c6ea2651a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:30:12.684867 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.684464    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-credential-socket" pod "6a80cae8-0b58-4643-aadb-771c6ea2651a" (UID: "6a80cae8-0b58-4643-aadb-771c6ea2651a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:30:12.684867 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.684495    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-workload-socket" pod "6a80cae8-0b58-4643-aadb-771c6ea2651a" (UID: "6a80cae8-0b58-4643-aadb-771c6ea2651a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:30:12.684867 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.684506    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a80cae8-0b58-4643-aadb-771c6ea2651a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a80cae8-0b58-4643-aadb-771c6ea2651a-istiod-ca-cert\") pod \"6a80cae8-0b58-4643-aadb-771c6ea2651a\" (UID: \"6a80cae8-0b58-4643-aadb-771c6ea2651a\") "
Oct 30 04:30:12.684867 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.684708    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-data" pod "6a80cae8-0b58-4643-aadb-771c6ea2651a" (UID: "6a80cae8-0b58-4643-aadb-771c6ea2651a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:30:12.684867 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.684840    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a80cae8-0b58-4643-aadb-771c6ea2651a-istiod-ca-cert" pod "6a80cae8-0b58-4643-aadb-771c6ea2651a" (UID: "6a80cae8-0b58-4643-aadb-771c6ea2651a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:30:12.685220 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.685176    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:30:12.685220 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.685198    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:30:12.685220 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.685212    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:30:12.685368 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.685225    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:30:12.685368 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.685254    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a80cae8-0b58-4643-aadb-771c6ea2651a-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:30:12.686195 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.686162    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-envoy" pod "6a80cae8-0b58-4643-aadb-771c6ea2651a" (UID: "6a80cae8-0b58-4643-aadb-771c6ea2651a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:30:12.686349 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.686329    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a80cae8-0b58-4643-aadb-771c6ea2651a-kube-api-access-jj8p9" pod "6a80cae8-0b58-4643-aadb-771c6ea2651a" (UID: "6a80cae8-0b58-4643-aadb-771c6ea2651a"). InnerVolumeSpecName "kube-api-access-jj8p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:30:12.687131 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.687111    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-podinfo" pod "6a80cae8-0b58-4643-aadb-771c6ea2651a" (UID: "6a80cae8-0b58-4643-aadb-771c6ea2651a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:30:12.687410 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.687394    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-token" pod "6a80cae8-0b58-4643-aadb-771c6ea2651a" (UID: "6a80cae8-0b58-4643-aadb-771c6ea2651a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:30:12.743526 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.741767    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"]
Oct 30 04:30:12.743526 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.742534    4874 cpu_manager.go:414] "containerMap: removing container" podUID="6a80cae8-0b58-4643-aadb-771c6ea2651a" containerName="istio-proxy"
Oct 30 04:30:12.743526 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.742551    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a80cae8-0b58-4643-aadb-771c6ea2651a" containerName="istio-proxy"
Oct 30 04:30:12.743526 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.742717    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a80cae8-0b58-4643-aadb-771c6ea2651a" containerName="istio-proxy"
Oct 30 04:30:12.749604 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.749414    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.772200 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.772179    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-153a44-openshift-default-dockercfg-b4w8s\""
Oct 30 04:30:12.786004 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.785966    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-credential-socket\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.786133 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.786078    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istiod-ca-cert\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.786133 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.786119    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-podinfo\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.786270 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.786169    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-envoy\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.786323 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.786292    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-workload-socket\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.786386 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.786326    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-token\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.786633 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.786584    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wlr6\" (UniqueName: \"kubernetes.io/projected/4ded57b2-1fa2-414f-9f3b-b598eee59dad-kube-api-access-8wlr6\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.786739 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.786642    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-workload-certs\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.786866 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.786711    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-data\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.786973 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.786929    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:30:12.786973 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.786947    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:30:12.786973 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.786960    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a80cae8-0b58-4643-aadb-771c6ea2651a-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:30:12.787192 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.786976    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj8p9\" (UniqueName: \"kubernetes.io/projected/6a80cae8-0b58-4643-aadb-771c6ea2651a-kube-api-access-jj8p9\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:30:12.789663 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.789636    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"]
Oct 30 04:30:12.887995 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.887912    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-envoy\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.888142 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.888003    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-workload-socket\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.888142 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.888031    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-token\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.888142 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.888104    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wlr6\" (UniqueName: \"kubernetes.io/projected/4ded57b2-1fa2-414f-9f3b-b598eee59dad-kube-api-access-8wlr6\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.888142 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.888130    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-workload-certs\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.888386 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.888151    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-data\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.888386 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.888202    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-credential-socket\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.888386 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.888311    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istiod-ca-cert\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.888386 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.888345    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-podinfo\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.888597 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.888449    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-workload-socket\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.888597 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.888516    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-data\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.888703 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.888619    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-workload-certs\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.888777 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.888756    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-credential-socket\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.889146 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.889122    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istiod-ca-cert\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.890910 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.890621    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-envoy\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.895811 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.891544    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-podinfo\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.974478 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.974411    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wlr6\" (UniqueName: \"kubernetes.io/projected/4ded57b2-1fa2-414f-9f3b-b598eee59dad-kube-api-access-8wlr6\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:12.975137 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:12.975086    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-token\") pod \"e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") " pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:13.067196 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:13.067165    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:13.133786 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:13.133760    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr"
Oct 30 04:30:13.260013 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:13.259981    4874 generic.go:356] "Generic (PLEG): container finished" podID="6a80cae8-0b58-4643-aadb-771c6ea2651a" containerID="85184c52773421c64459159dfaf6c0029b07dce151dca6d02948879f987788f4" exitCode=0
Oct 30 04:30:13.260204 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:13.260028    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"
Oct 30 04:30:13.260204 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:13.260060    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh" event={"ID":"6a80cae8-0b58-4643-aadb-771c6ea2651a","Type":"ContainerDied","Data":"85184c52773421c64459159dfaf6c0029b07dce151dca6d02948879f987788f4"}
Oct 30 04:30:13.260204 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:13.260107    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh" event={"ID":"6a80cae8-0b58-4643-aadb-771c6ea2651a","Type":"ContainerDied","Data":"037cc8165f0bcd62a00d27c073fba4086d399a0290aa4f3ad867d986862dc05a"}
Oct 30 04:30:13.260204 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:13.260129    4874 scope.go:122] "RemoveContainer" containerID="85184c52773421c64459159dfaf6c0029b07dce151dca6d02948879f987788f4"
Oct 30 04:30:13.261889 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:13.261863    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm" event={"ID":"d8a444bc-289c-49df-a4e0-e85a241e73dc","Type":"ContainerStarted","Data":"5aba1f04b2208c8f7933499d998ccb43fa94fc8f41b0707a9b711b83662693c8"}
Oct 30 04:30:13.262003 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:13.261916    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm"
Oct 30 04:30:13.269810 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:13.269795    4874 scope.go:122] "RemoveContainer" containerID="85184c52773421c64459159dfaf6c0029b07dce151dca6d02948879f987788f4"
Oct 30 04:30:13.270040 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:30:13.270021    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85184c52773421c64459159dfaf6c0029b07dce151dca6d02948879f987788f4\": container with ID starting with 85184c52773421c64459159dfaf6c0029b07dce151dca6d02948879f987788f4 not found: ID does not exist" containerID="85184c52773421c64459159dfaf6c0029b07dce151dca6d02948879f987788f4"
Oct 30 04:30:13.270090 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:13.270047    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85184c52773421c64459159dfaf6c0029b07dce151dca6d02948879f987788f4"} err="failed to get container status \"85184c52773421c64459159dfaf6c0029b07dce151dca6d02948879f987788f4\": rpc error: code = NotFound desc = could not find container \"85184c52773421c64459159dfaf6c0029b07dce151dca6d02948879f987788f4\": container with ID starting with 85184c52773421c64459159dfaf6c0029b07dce151dca6d02948879f987788f4 not found: ID does not exist"
Oct 30 04:30:13.309718 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:13.309651    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"]
Oct 30 04:30:13.321091 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:13.321065    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-zqvhh"]
Oct 30 04:30:13.352653 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:13.352590    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm" podStartSLOduration=2.949354449 podStartE2EDuration="3.352572645s" totalImagesPullingTime="403.218196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:13.351502657 +0000 UTC m=+1932.763898231" watchObservedRunningTime="2026-10-30 04:30:13.352572645 +0000 UTC m=+1932.764968216"
Oct 30 04:30:13.387704 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:13.387684    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"]
Oct 30 04:30:13.389721 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:30:13.389689    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ded57b2_1fa2_414f_9f3b_b598eee59dad.slice/crio-e75aea795baf42b63466af7684c022d6ecf28004fba0a80b956c7949a7e8d8f9 WatchSource:0}: Error finding container e75aea795baf42b63466af7684c022d6ecf28004fba0a80b956c7949a7e8d8f9: Status 404 returned error can't find the container with id e75aea795baf42b63466af7684c022d6ecf28004fba0a80b956c7949a7e8d8f9
Oct 30 04:30:13.392434 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:13.392403    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:30:13.392522 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:13.392477    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:30:13.392558 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:13.392518    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:30:14.137598 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:14.137572    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-71b161-668476b448-8489m"
Oct 30 04:30:14.269619 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:14.269579    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp" event={"ID":"4ded57b2-1fa2-414f-9f3b-b598eee59dad","Type":"ContainerStarted","Data":"04e566564f84cf9972937d9e23690af8b7698c68c76bb00f9d86c254a07bef16"}
Oct 30 04:30:14.269761 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:14.269641    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp" event={"ID":"4ded57b2-1fa2-414f-9f3b-b598eee59dad","Type":"ContainerStarted","Data":"e75aea795baf42b63466af7684c022d6ecf28004fba0a80b956c7949a7e8d8f9"}
Oct 30 04:30:14.321816 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:14.321755    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp" podStartSLOduration=2.321738548 podStartE2EDuration="2.321738548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:14.32125733 +0000 UTC m=+1933.733652921" watchObservedRunningTime="2026-10-30 04:30:14.321738548 +0000 UTC m=+1933.734134121"
Oct 30 04:30:14.741141 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:14.741105    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a80cae8-0b58-4643-aadb-771c6ea2651a" path="/var/lib/kubelet/pods/6a80cae8-0b58-4643-aadb-771c6ea2651a/volumes"
Oct 30 04:30:15.067788 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:15.067711    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:15.072758 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:15.072734    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:15.275639 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:15.275609    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:15.276438 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:15.276421    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:30:18.232194 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:18.232155    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj"
Oct 30 04:30:18.654446 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:18.654367    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll"]
Oct 30 04:30:18.659410 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:18.659391    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll"
Oct 30 04:30:18.685577 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:18.685542    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll"]
Oct 30 04:30:18.685728 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:18.685704    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-153a44\""
Oct 30 04:30:18.685783 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:18.685759    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-153a44-dockercfg-5kb2b\""
Oct 30 04:30:18.806888 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:18.806854    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vnw5\" (UniqueName: \"kubernetes.io/projected/744645d7-7974-4014-b58e-3129785027ad-kube-api-access-6vnw5\") pod \"payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll\" (UID: \"744645d7-7974-4014-b58e-3129785027ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll"
Oct 30 04:30:18.807312 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:18.807289    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744645d7-7974-4014-b58e-3129785027ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll\" (UID: \"744645d7-7974-4014-b58e-3129785027ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll"
Oct 30 04:30:18.908426 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:18.908330    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744645d7-7974-4014-b58e-3129785027ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll\" (UID: \"744645d7-7974-4014-b58e-3129785027ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll"
Oct 30 04:30:18.908600 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:18.908494    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vnw5\" (UniqueName: \"kubernetes.io/projected/744645d7-7974-4014-b58e-3129785027ad-kube-api-access-6vnw5\") pod \"payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll\" (UID: \"744645d7-7974-4014-b58e-3129785027ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll"
Oct 30 04:30:18.909005 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:18.908985    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744645d7-7974-4014-b58e-3129785027ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll\" (UID: \"744645d7-7974-4014-b58e-3129785027ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll"
Oct 30 04:30:18.939383 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:18.939355    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vnw5\" (UniqueName: \"kubernetes.io/projected/744645d7-7974-4014-b58e-3129785027ad-kube-api-access-6vnw5\") pod \"payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll\" (UID: \"744645d7-7974-4014-b58e-3129785027ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll"
Oct 30 04:30:18.972048 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:18.972017    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll"
Oct 30 04:30:19.275496 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:19.275458    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm"
Oct 30 04:30:19.277906 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:19.277884    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll"]
Oct 30 04:30:19.280228 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:30:19.280204    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744645d7_7974_4014_b58e_3129785027ad.slice/crio-0cb8306c6382598828d74c3ed84f5b2eb7cf11d56ed94ccd05619a7e5b8b8164 WatchSource:0}: Error finding container 0cb8306c6382598828d74c3ed84f5b2eb7cf11d56ed94ccd05619a7e5b8b8164: Status 404 returned error can't find the container with id 0cb8306c6382598828d74c3ed84f5b2eb7cf11d56ed94ccd05619a7e5b8b8164
Oct 30 04:30:19.299494 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:19.299470    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll" event={"ID":"744645d7-7974-4014-b58e-3129785027ad","Type":"ContainerStarted","Data":"0cb8306c6382598828d74c3ed84f5b2eb7cf11d56ed94ccd05619a7e5b8b8164"}
Oct 30 04:30:20.243532 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:20.243505    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6"
Oct 30 04:30:20.309529 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:20.309485    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll" event={"ID":"744645d7-7974-4014-b58e-3129785027ad","Type":"ContainerStarted","Data":"a5f21fe0ef49091e862150b4b2bd59cee0e66310435d704387bc780a19324c7e"}
Oct 30 04:30:20.310001 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:20.309681    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll"
Oct 30 04:30:20.350275 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:20.350202    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll" podStartSLOduration=2.350177419 podStartE2EDuration="2.350177419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:20.347536473 +0000 UTC m=+1939.759932046" watchObservedRunningTime="2026-10-30 04:30:20.350177419 +0000 UTC m=+1939.762572992"
Oct 30 04:30:24.390935 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.390901    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"]
Oct 30 04:30:24.397026 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.396929    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.404727 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.404689    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-credential-socket\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.405046 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.405016    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istiod-ca-cert\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.405155 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.405115    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-envoy\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.405224 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.405193    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58hcq\" (UniqueName: \"kubernetes.io/projected/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-kube-api-access-58hcq\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.405318 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.405274    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-workload-socket\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.405443 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.405426    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-data\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.405688 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.405637    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-podinfo\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.405871 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.405839    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-workload-certs\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.406049 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.406029    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-token\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.413422 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.413377    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-6a777c-openshift-default-dockercfg-hjlzj\""
Oct 30 04:30:24.427385 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.427359    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"]
Oct 30 04:30:24.511615 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.511450    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istiod-ca-cert\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.514788 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.514747    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istiod-ca-cert\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.515044 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.515024    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-envoy\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.516050 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.516027    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58hcq\" (UniqueName: \"kubernetes.io/projected/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-kube-api-access-58hcq\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.516455 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.516435    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-workload-socket\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.516918 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.516817    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-workload-socket\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.517340 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.517319    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-data\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.518549 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.518527    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-podinfo\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.519467 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.519447    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-workload-certs\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.519918 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.518959    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-data\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.520097 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.519810    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-workload-certs\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.520284 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.520263    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-token\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.520476 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.520457    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-credential-socket\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.520950 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.520933    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-credential-socket\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.600262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.593510    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-podinfo\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.600262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.593750    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-envoy\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.618259 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.609804    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-token\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.625091 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.625058    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58hcq\" (UniqueName: \"kubernetes.io/projected/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-kube-api-access-58hcq\") pod \"e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") " pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:24.715784 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:24.715460    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:25.090071 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:25.090024    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"]
Oct 30 04:30:25.094536 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:30:25.094510    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0cc4fd_bf2b_4642_ab23_e6649237f0fb.slice/crio-d230364bce23af75a4ffcf47c41683e7aab5456a8ef361aa52566ee9dd17c235 WatchSource:0}: Error finding container d230364bce23af75a4ffcf47c41683e7aab5456a8ef361aa52566ee9dd17c235: Status 404 returned error can't find the container with id d230364bce23af75a4ffcf47c41683e7aab5456a8ef361aa52566ee9dd17c235
Oct 30 04:30:25.097466 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:25.097436    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:30:25.097547 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:25.097503    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:30:25.097547 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:25.097530    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:30:25.340234 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:25.340148    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v" event={"ID":"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb","Type":"ContainerStarted","Data":"f640d6169b90ad661207d8685aba0f6f09fdfe54c9f5e0896727909458157434"}
Oct 30 04:30:25.340234 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:25.340209    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v" event={"ID":"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb","Type":"ContainerStarted","Data":"d230364bce23af75a4ffcf47c41683e7aab5456a8ef361aa52566ee9dd17c235"}
Oct 30 04:30:25.386137 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:25.386062    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v" podStartSLOduration=1.386042642 podStartE2EDuration="1.386042642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:25.383699109 +0000 UTC m=+1944.796094683" watchObservedRunningTime="2026-10-30 04:30:25.386042642 +0000 UTC m=+1944.798438296"
Oct 30 04:30:25.623540 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:30:25.623462    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:30:25.716514 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:25.716474    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:26.729970 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:26.729943    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:27.359969 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:27.359927    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:27.361148 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:27.361125    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:30:31.319543 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:31.319518    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll"
Oct 30 04:30:32.390793 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:32.390756    4874 generic.go:356] "Generic (PLEG): container finished" podID="6a69deca-d8cf-4ea4-bb1f-15bee6a789a4" containerID="220354a581ef639ab086606c6b464cc89579d421dae2d836bc820fd532dc51a8" exitCode=0
Oct 30 04:30:32.391211 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:32.390828    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s" event={"ID":"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4","Type":"ContainerDied","Data":"220354a581ef639ab086606c6b464cc89579d421dae2d836bc820fd532dc51a8"}
Oct 30 04:30:32.391211 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:32.391207    4874 scope.go:122] "RemoveContainer" containerID="220354a581ef639ab086606c6b464cc89579d421dae2d836bc820fd532dc51a8"
Oct 30 04:30:32.392687 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:32.392663    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:30:32.392753 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:32.392727    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:30:32.392790 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:32.392758    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:30:33.398711 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:33.398677    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s" event={"ID":"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4","Type":"ContainerStarted","Data":"26fd51259d1c410f7211180d96dac6bc4614dcb74b50bb50e7fabaa855b90f14"}
Oct 30 04:30:33.516560 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:33.516524    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:30:33.517673 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:33.517646    4874 patch_prober.go:29] interesting pod/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 30 04:30:33.517793 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:33.517702    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s" podUID="6a69deca-d8cf-4ea4-bb1f-15bee6a789a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 30 04:30:34.516741 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:34.516709    4874 patch_prober.go:29] interesting pod/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 30 04:30:34.517131 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:34.516769    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s" podUID="6a69deca-d8cf-4ea4-bb1f-15bee6a789a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 30 04:30:35.517324 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:35.517287    4874 patch_prober.go:29] interesting pod/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 30 04:30:35.517788 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:35.517355    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s" podUID="6a69deca-d8cf-4ea4-bb1f-15bee6a789a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 30 04:30:36.517428 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:36.517389    4874 patch_prober.go:29] interesting pod/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 30 04:30:36.517907 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:36.517464    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s" podUID="6a69deca-d8cf-4ea4-bb1f-15bee6a789a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 30 04:30:36.989472 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:36.989363    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:30:37.436296 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:37.436192    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:30:37.522694 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:37.522659    4874 patch_prober.go:29] interesting pod/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 30 04:30:37.523332 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:37.523299    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s" podUID="6a69deca-d8cf-4ea4-bb1f-15bee6a789a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 30 04:30:38.089936 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:38.089903    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz"]
Oct 30 04:30:38.094099 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:38.094066    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz"
Oct 30 04:30:38.097964 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:38.097649    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz"]
Oct 30 04:30:38.149954 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:38.149920    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-153a44-serving-cert\""
Oct 30 04:30:38.228264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:38.198662    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8lwq\" (UniqueName: \"kubernetes.io/projected/ef50e538-692c-4669-8861-def5ea47dc1f-kube-api-access-c8lwq\") pod \"maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz\" (UID: \"ef50e538-692c-4669-8861-def5ea47dc1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz"
Oct 30 04:30:38.246277 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:38.242191    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef50e538-692c-4669-8861-def5ea47dc1f-metrics-certs\") pod \"maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz\" (UID: \"ef50e538-692c-4669-8861-def5ea47dc1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz"
Oct 30 04:30:38.246277 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:38.242571    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef50e538-692c-4669-8861-def5ea47dc1f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz\" (UID: \"ef50e538-692c-4669-8861-def5ea47dc1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz"
Oct 30 04:30:38.246277 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:38.242793    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef50e538-692c-4669-8861-def5ea47dc1f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz\" (UID: \"ef50e538-692c-4669-8861-def5ea47dc1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz"
Oct 30 04:30:38.344552 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:38.344454    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8lwq\" (UniqueName: \"kubernetes.io/projected/ef50e538-692c-4669-8861-def5ea47dc1f-kube-api-access-c8lwq\") pod \"maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz\" (UID: \"ef50e538-692c-4669-8861-def5ea47dc1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz"
Oct 30 04:30:38.344552 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:38.344545    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef50e538-692c-4669-8861-def5ea47dc1f-metrics-certs\") pod \"maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz\" (UID: \"ef50e538-692c-4669-8861-def5ea47dc1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz"
Oct 30 04:30:38.344813 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:38.344586    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef50e538-692c-4669-8861-def5ea47dc1f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz\" (UID: \"ef50e538-692c-4669-8861-def5ea47dc1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz"
Oct 30 04:30:38.344813 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:38.344693    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef50e538-692c-4669-8861-def5ea47dc1f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz\" (UID: \"ef50e538-692c-4669-8861-def5ea47dc1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz"
Oct 30 04:30:38.345588 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:38.345559    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef50e538-692c-4669-8861-def5ea47dc1f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz\" (UID: \"ef50e538-692c-4669-8861-def5ea47dc1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz"
Oct 30 04:30:38.347082 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:38.347059    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef50e538-692c-4669-8861-def5ea47dc1f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz\" (UID: \"ef50e538-692c-4669-8861-def5ea47dc1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz"
Oct 30 04:30:38.347276 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:38.347252    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef50e538-692c-4669-8861-def5ea47dc1f-metrics-certs\") pod \"maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz\" (UID: \"ef50e538-692c-4669-8861-def5ea47dc1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz"
Oct 30 04:30:38.366486 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:38.366464    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8lwq\" (UniqueName: \"kubernetes.io/projected/ef50e538-692c-4669-8861-def5ea47dc1f-kube-api-access-c8lwq\") pod \"maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz\" (UID: \"ef50e538-692c-4669-8861-def5ea47dc1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz"
Oct 30 04:30:38.550623 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:38.550589    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz"
Oct 30 04:30:38.789296 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:38.789265    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:30:38.790572 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:38.790546    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:30:38.854687 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:38.854616    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz"]
Oct 30 04:30:38.857140 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:30:38.857115    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef50e538_692c_4669_8861_def5ea47dc1f.slice/crio-faffcb13b7c865c79692054da368c42e8e6d7064878c4678b159692f4dec6323 WatchSource:0}: Error finding container faffcb13b7c865c79692054da368c42e8e6d7064878c4678b159692f4dec6323: Status 404 returned error can't find the container with id faffcb13b7c865c79692054da368c42e8e6d7064878c4678b159692f4dec6323
Oct 30 04:30:39.378481 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:39.378392    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9"]
Oct 30 04:30:39.478618 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:39.478569    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz" event={"ID":"ef50e538-692c-4669-8861-def5ea47dc1f","Type":"ContainerStarted","Data":"1850991bf5dcacbddf09c78c725d8cae960ba34cfd460b5f129d8fac1aabc34e"}
Oct 30 04:30:39.478934 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:39.478893    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz" event={"ID":"ef50e538-692c-4669-8861-def5ea47dc1f","Type":"ContainerStarted","Data":"faffcb13b7c865c79692054da368c42e8e6d7064878c4678b159692f4dec6323"}
Oct 30 04:30:39.479084 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:39.478951    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz"
Oct 30 04:30:44.378810 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:44.378769    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9" podUID="ecd33009-bafc-4f73-a476-f5d852966202" containerName="payload-processing" containerID="cri-o://4df9644768e08626e7827367fec3f53c93443484c81d1a5df48c0c57773da15f" gracePeriod=25
Oct 30 04:30:44.511184 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:44.508371    4874 generic.go:356] "Generic (PLEG): container finished" podID="ecd33009-bafc-4f73-a476-f5d852966202" containerID="4df9644768e08626e7827367fec3f53c93443484c81d1a5df48c0c57773da15f" exitCode=0
Oct 30 04:30:44.511184 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:44.508926    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9" event={"ID":"ecd33009-bafc-4f73-a476-f5d852966202","Type":"ContainerDied","Data":"4df9644768e08626e7827367fec3f53c93443484c81d1a5df48c0c57773da15f"}
Oct 30 04:30:44.635945 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:44.635889    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9"
Oct 30 04:30:44.675737 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:44.675675    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz" podStartSLOduration=6.4332516 podStartE2EDuration="6.675656315s" totalImagesPullingTime="242.404715ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:39.521490929 +0000 UTC m=+1958.933886504" watchObservedRunningTime="2026-10-30 04:30:44.675656315 +0000 UTC m=+1964.088051890"
Oct 30 04:30:44.800686 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:44.800651    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecd33009-bafc-4f73-a476-f5d852966202-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd33009-bafc-4f73-a476-f5d852966202-plugins-config-volume\") pod \"ecd33009-bafc-4f73-a476-f5d852966202\" (UID: \"ecd33009-bafc-4f73-a476-f5d852966202\") "
Oct 30 04:30:44.800870 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:44.800851    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecd33009-bafc-4f73-a476-f5d852966202-kube-api-access-t6bdm\" (UniqueName: \"kubernetes.io/projected/ecd33009-bafc-4f73-a476-f5d852966202-kube-api-access-t6bdm\") pod \"ecd33009-bafc-4f73-a476-f5d852966202\" (UID: \"ecd33009-bafc-4f73-a476-f5d852966202\") "
Oct 30 04:30:44.801037 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:44.801012    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd33009-bafc-4f73-a476-f5d852966202-plugins-config-volume" pod "ecd33009-bafc-4f73-a476-f5d852966202" (UID: "ecd33009-bafc-4f73-a476-f5d852966202"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:30:44.801299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:44.801279    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd33009-bafc-4f73-a476-f5d852966202-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:30:44.803336 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:44.803305    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd33009-bafc-4f73-a476-f5d852966202-kube-api-access-t6bdm" pod "ecd33009-bafc-4f73-a476-f5d852966202" (UID: "ecd33009-bafc-4f73-a476-f5d852966202"). InnerVolumeSpecName "kube-api-access-t6bdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:30:44.902426 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:44.902353    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6bdm\" (UniqueName: \"kubernetes.io/projected/ecd33009-bafc-4f73-a476-f5d852966202-kube-api-access-t6bdm\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:30:45.489458 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:45.489429    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz"
Oct 30 04:30:45.515610 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:45.515571    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9" event={"ID":"ecd33009-bafc-4f73-a476-f5d852966202","Type":"ContainerDied","Data":"26ec6bbf42ab11c94e143537b5c5ae4c9ca2314dd0da6a6755ef62941dc3f709"}
Oct 30 04:30:45.515610 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:45.515596    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9"
Oct 30 04:30:45.515797 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:45.515630    4874 scope.go:122] "RemoveContainer" containerID="4df9644768e08626e7827367fec3f53c93443484c81d1a5df48c0c57773da15f"
Oct 30 04:30:45.566127 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:45.566091    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9"]
Oct 30 04:30:45.569842 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:45.569812    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dbabed-5d5d4d9966-nxqp9"]
Oct 30 04:30:46.728997 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:46.728965    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecd33009-bafc-4f73-a476-f5d852966202" path="/var/lib/kubelet/pods/ecd33009-bafc-4f73-a476-f5d852966202/volumes"
Oct 30 04:30:52.163868 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:52.163836    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn"]
Oct 30 04:30:52.164383 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:52.164369    4874 cpu_manager.go:414] "containerMap: removing container" podUID="ecd33009-bafc-4f73-a476-f5d852966202" containerName="payload-processing"
Oct 30 04:30:52.164430 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:52.164384    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd33009-bafc-4f73-a476-f5d852966202" containerName="payload-processing"
Oct 30 04:30:52.164483 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:52.164474    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd33009-bafc-4f73-a476-f5d852966202" containerName="payload-processing"
Oct 30 04:30:52.168962 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:52.168946    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn"
Oct 30 04:30:52.189122 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:52.189099    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-6a777c-serving-cert\""
Oct 30 04:30:52.196924 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:52.196894    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d8911a9-d86c-4efb-99dc-156a74ad8628-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn\" (UID: \"3d8911a9-d86c-4efb-99dc-156a74ad8628\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn"
Oct 30 04:30:52.197072 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:52.196967    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8911a9-d86c-4efb-99dc-156a74ad8628-metrics-certs\") pod \"maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn\" (UID: \"3d8911a9-d86c-4efb-99dc-156a74ad8628\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn"
Oct 30 04:30:52.197444 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:52.197105    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2ln7\" (UniqueName: \"kubernetes.io/projected/3d8911a9-d86c-4efb-99dc-156a74ad8628-kube-api-access-p2ln7\") pod \"maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn\" (UID: \"3d8911a9-d86c-4efb-99dc-156a74ad8628\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn"
Oct 30 04:30:52.197444 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:52.197179    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d8911a9-d86c-4efb-99dc-156a74ad8628-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn\" (UID: \"3d8911a9-d86c-4efb-99dc-156a74ad8628\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn"
Oct 30 04:30:52.205068 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:52.205041    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn"]
Oct 30 04:30:52.298638 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:52.298595    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d8911a9-d86c-4efb-99dc-156a74ad8628-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn\" (UID: \"3d8911a9-d86c-4efb-99dc-156a74ad8628\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn"
Oct 30 04:30:52.298826 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:52.298671    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8911a9-d86c-4efb-99dc-156a74ad8628-metrics-certs\") pod \"maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn\" (UID: \"3d8911a9-d86c-4efb-99dc-156a74ad8628\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn"
Oct 30 04:30:52.298826 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:30:52.298718    4874 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-serving-cert: secret "maas-api-e2e-ipp-b-6a777c-serving-cert" not found
Oct 30 04:30:52.298826 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:30:52.298792    4874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8911a9-d86c-4efb-99dc-156a74ad8628-maas-api-tls podName:3d8911a9-d86c-4efb-99dc-156a74ad8628 nodeName:}" failed. No retries permitted until 2026-10-30 04:30:52.798775593 +0000 UTC m=+1972.211171147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3d8911a9-d86c-4efb-99dc-156a74ad8628-maas-api-tls") pod "maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn" (UID: "3d8911a9-d86c-4efb-99dc-156a74ad8628") : secret "maas-api-e2e-ipp-b-6a777c-serving-cert" not found
Oct 30 04:30:52.298826 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:52.298822    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2ln7\" (UniqueName: \"kubernetes.io/projected/3d8911a9-d86c-4efb-99dc-156a74ad8628-kube-api-access-p2ln7\") pod \"maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn\" (UID: \"3d8911a9-d86c-4efb-99dc-156a74ad8628\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn"
Oct 30 04:30:52.299061 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:52.298898    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d8911a9-d86c-4efb-99dc-156a74ad8628-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn\" (UID: \"3d8911a9-d86c-4efb-99dc-156a74ad8628\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn"
Oct 30 04:30:52.299546 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:52.299524    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d8911a9-d86c-4efb-99dc-156a74ad8628-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn\" (UID: \"3d8911a9-d86c-4efb-99dc-156a74ad8628\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn"
Oct 30 04:30:52.301079 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:52.301056    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8911a9-d86c-4efb-99dc-156a74ad8628-metrics-certs\") pod \"maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn\" (UID: \"3d8911a9-d86c-4efb-99dc-156a74ad8628\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn"
Oct 30 04:30:52.321590 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:52.321562    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2ln7\" (UniqueName: \"kubernetes.io/projected/3d8911a9-d86c-4efb-99dc-156a74ad8628-kube-api-access-p2ln7\") pod \"maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn\" (UID: \"3d8911a9-d86c-4efb-99dc-156a74ad8628\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn"
Oct 30 04:30:52.807723 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:52.807676    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d8911a9-d86c-4efb-99dc-156a74ad8628-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn\" (UID: \"3d8911a9-d86c-4efb-99dc-156a74ad8628\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn"
Oct 30 04:30:52.817532 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:52.817501    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d8911a9-d86c-4efb-99dc-156a74ad8628-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn\" (UID: \"3d8911a9-d86c-4efb-99dc-156a74ad8628\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn"
Oct 30 04:30:52.970510 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:52.970476    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"]
Oct 30 04:30:52.970780 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:52.970737    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s" podUID="4e716825-de92-4c9d-9b36-007ea4218a98" containerName="istio-proxy" containerID="cri-o://e184a4f17326316b5f030bfa193458ae7d9a9d0d8aafb46449a52886b23fb315" gracePeriod=30
Oct 30 04:30:53.082550 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:53.082474    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn"
Oct 30 04:30:53.469594 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:53.469561    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn"]
Oct 30 04:30:53.471424 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:30:53.471394    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8911a9_d86c_4efb_99dc_156a74ad8628.slice/crio-d0c20c1c310a503a0e10df7f56566c8284470389693fb23438108a3248533d16 WatchSource:0}: Error finding container d0c20c1c310a503a0e10df7f56566c8284470389693fb23438108a3248533d16: Status 404 returned error can't find the container with id d0c20c1c310a503a0e10df7f56566c8284470389693fb23438108a3248533d16
Oct 30 04:30:53.564187 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:53.564155    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn" event={"ID":"3d8911a9-d86c-4efb-99dc-156a74ad8628","Type":"ContainerStarted","Data":"d0c20c1c310a503a0e10df7f56566c8284470389693fb23438108a3248533d16"}
Oct 30 04:30:54.571932 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:54.571899    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn" event={"ID":"3d8911a9-d86c-4efb-99dc-156a74ad8628","Type":"ContainerStarted","Data":"d71e182b92a06babbb089ba9d3c2550cae60410eafcaa8029a37ae556411295c"}
Oct 30 04:30:54.572472 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:54.572028    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn"
Oct 30 04:30:54.617179 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:54.617134    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn" podStartSLOduration=2.219184086 podStartE2EDuration="2.617120907s" totalImagesPullingTime="397.936821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:54.616706712 +0000 UTC m=+1974.029102286" watchObservedRunningTime="2026-10-30 04:30:54.617120907 +0000 UTC m=+1974.029516482"
Oct 30 04:30:55.612301 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:30:55.612260    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:30:56.195390 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:56.195356    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 04:30:56.195652 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:56.195624    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s" podUID="4e716825-de92-4c9d-9b36-007ea4218a98" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:30:58.224652 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.224630    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:30:58.323419 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.323347    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e716825-de92-4c9d-9b36-007ea4218a98-kube-api-access-t9whx\" (UniqueName: \"kubernetes.io/projected/4e716825-de92-4c9d-9b36-007ea4218a98-kube-api-access-t9whx\") pod \"4e716825-de92-4c9d-9b36-007ea4218a98\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") "
Oct 30 04:30:58.323419 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.323399    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-credential-socket\") pod \"4e716825-de92-4c9d-9b36-007ea4218a98\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") "
Oct 30 04:30:58.323610 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.323435    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e716825-de92-4c9d-9b36-007ea4218a98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e716825-de92-4c9d-9b36-007ea4218a98-istio-podinfo\") pod \"4e716825-de92-4c9d-9b36-007ea4218a98\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") "
Oct 30 04:30:58.323610 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.323471    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-istio-data\") pod \"4e716825-de92-4c9d-9b36-007ea4218a98\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") "
Oct 30 04:30:58.323610 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.323498    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e716825-de92-4c9d-9b36-007ea4218a98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e716825-de92-4c9d-9b36-007ea4218a98-istiod-ca-cert\") pod \"4e716825-de92-4c9d-9b36-007ea4218a98\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") "
Oct 30 04:30:58.323610 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.323530    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-workload-certs\") pod \"4e716825-de92-4c9d-9b36-007ea4218a98\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") "
Oct 30 04:30:58.323812 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.323660    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e716825-de92-4c9d-9b36-007ea4218a98-istio-token\" (UniqueName: \"kubernetes.io/projected/4e716825-de92-4c9d-9b36-007ea4218a98-istio-token\") pod \"4e716825-de92-4c9d-9b36-007ea4218a98\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") "
Oct 30 04:30:58.323812 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.323762    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-workload-socket\") pod \"4e716825-de92-4c9d-9b36-007ea4218a98\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") "
Oct 30 04:30:58.323812 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.323807    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-istio-envoy\") pod \"4e716825-de92-4c9d-9b36-007ea4218a98\" (UID: \"4e716825-de92-4c9d-9b36-007ea4218a98\") "
Oct 30 04:30:58.323973 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.323837    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-credential-socket" pod "4e716825-de92-4c9d-9b36-007ea4218a98" (UID: "4e716825-de92-4c9d-9b36-007ea4218a98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:30:58.323973 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.323884    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-istio-data" pod "4e716825-de92-4c9d-9b36-007ea4218a98" (UID: "4e716825-de92-4c9d-9b36-007ea4218a98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:30:58.324284 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.324091    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-workload-socket" pod "4e716825-de92-4c9d-9b36-007ea4218a98" (UID: "4e716825-de92-4c9d-9b36-007ea4218a98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:30:58.324284 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.324225    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e716825-de92-4c9d-9b36-007ea4218a98-istiod-ca-cert" pod "4e716825-de92-4c9d-9b36-007ea4218a98" (UID: "4e716825-de92-4c9d-9b36-007ea4218a98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:30:58.324491 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.324412    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:30:58.324491 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.324437    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:30:58.324491 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.324452    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e716825-de92-4c9d-9b36-007ea4218a98-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:30:58.324491 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.324466    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:30:58.324846 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.324691    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-workload-certs" pod "4e716825-de92-4c9d-9b36-007ea4218a98" (UID: "4e716825-de92-4c9d-9b36-007ea4218a98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:30:58.325748 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.325720    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e716825-de92-4c9d-9b36-007ea4218a98-kube-api-access-t9whx" pod "4e716825-de92-4c9d-9b36-007ea4218a98" (UID: "4e716825-de92-4c9d-9b36-007ea4218a98"). InnerVolumeSpecName "kube-api-access-t9whx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:30:58.326122 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.326103    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-istio-envoy" pod "4e716825-de92-4c9d-9b36-007ea4218a98" (UID: "4e716825-de92-4c9d-9b36-007ea4218a98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:30:58.326327 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.326307    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e716825-de92-4c9d-9b36-007ea4218a98-istio-podinfo" pod "4e716825-de92-4c9d-9b36-007ea4218a98" (UID: "4e716825-de92-4c9d-9b36-007ea4218a98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:30:58.326376 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.326314    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e716825-de92-4c9d-9b36-007ea4218a98-istio-token" pod "4e716825-de92-4c9d-9b36-007ea4218a98" (UID: "4e716825-de92-4c9d-9b36-007ea4218a98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:30:58.425814 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.425780    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e716825-de92-4c9d-9b36-007ea4218a98-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:30:58.425814 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.425809    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:30:58.425814 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.425819    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e716825-de92-4c9d-9b36-007ea4218a98-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:30:58.426037 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.425826    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e716825-de92-4c9d-9b36-007ea4218a98-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:30:58.426037 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.425835    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9whx\" (UniqueName: \"kubernetes.io/projected/4e716825-de92-4c9d-9b36-007ea4218a98-kube-api-access-t9whx\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:30:58.597965 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.597881    4874 generic.go:356] "Generic (PLEG): container finished" podID="4e716825-de92-4c9d-9b36-007ea4218a98" containerID="e184a4f17326316b5f030bfa193458ae7d9a9d0d8aafb46449a52886b23fb315" exitCode=0
Oct 30 04:30:58.597965 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.597937    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"
Oct 30 04:30:58.598195 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.597971    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s" event={"ID":"4e716825-de92-4c9d-9b36-007ea4218a98","Type":"ContainerDied","Data":"e184a4f17326316b5f030bfa193458ae7d9a9d0d8aafb46449a52886b23fb315"}
Oct 30 04:30:58.598195 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.598055    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s" event={"ID":"4e716825-de92-4c9d-9b36-007ea4218a98","Type":"ContainerDied","Data":"6067be3afe85426ba4618bd66f8cb35bcc5e75c7a8b6a2adc391a5fd7a3d89c7"}
Oct 30 04:30:58.598195 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.598084    4874 scope.go:122] "RemoveContainer" containerID="e184a4f17326316b5f030bfa193458ae7d9a9d0d8aafb46449a52886b23fb315"
Oct 30 04:30:58.608082 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.608067    4874 scope.go:122] "RemoveContainer" containerID="2f09cbdac2cbd04e842944626106cc89e120926fdef07ae3a2074d673c42284d"
Oct 30 04:30:58.617049 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.617028    4874 scope.go:122] "RemoveContainer" containerID="e184a4f17326316b5f030bfa193458ae7d9a9d0d8aafb46449a52886b23fb315"
Oct 30 04:30:58.617317 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:30:58.617297    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e184a4f17326316b5f030bfa193458ae7d9a9d0d8aafb46449a52886b23fb315\": container with ID starting with e184a4f17326316b5f030bfa193458ae7d9a9d0d8aafb46449a52886b23fb315 not found: ID does not exist" containerID="e184a4f17326316b5f030bfa193458ae7d9a9d0d8aafb46449a52886b23fb315"
Oct 30 04:30:58.617387 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.617325    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e184a4f17326316b5f030bfa193458ae7d9a9d0d8aafb46449a52886b23fb315"} err="failed to get container status \"e184a4f17326316b5f030bfa193458ae7d9a9d0d8aafb46449a52886b23fb315\": rpc error: code = NotFound desc = could not find container \"e184a4f17326316b5f030bfa193458ae7d9a9d0d8aafb46449a52886b23fb315\": container with ID starting with e184a4f17326316b5f030bfa193458ae7d9a9d0d8aafb46449a52886b23fb315 not found: ID does not exist"
Oct 30 04:30:58.617387 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.617340    4874 scope.go:122] "RemoveContainer" containerID="2f09cbdac2cbd04e842944626106cc89e120926fdef07ae3a2074d673c42284d"
Oct 30 04:30:58.617549 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:30:58.617532    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f09cbdac2cbd04e842944626106cc89e120926fdef07ae3a2074d673c42284d\": container with ID starting with 2f09cbdac2cbd04e842944626106cc89e120926fdef07ae3a2074d673c42284d not found: ID does not exist" containerID="2f09cbdac2cbd04e842944626106cc89e120926fdef07ae3a2074d673c42284d"
Oct 30 04:30:58.617603 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.617553    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f09cbdac2cbd04e842944626106cc89e120926fdef07ae3a2074d673c42284d"} err="failed to get container status \"2f09cbdac2cbd04e842944626106cc89e120926fdef07ae3a2074d673c42284d\": rpc error: code = NotFound desc = could not find container \"2f09cbdac2cbd04e842944626106cc89e120926fdef07ae3a2074d673c42284d\": container with ID starting with 2f09cbdac2cbd04e842944626106cc89e120926fdef07ae3a2074d673c42284d not found: ID does not exist"
Oct 30 04:30:58.648407 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.648381    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"]
Oct 30 04:30:58.654136 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.654117    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-gr45s"]
Oct 30 04:30:58.730055 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:58.730026    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e716825-de92-4c9d-9b36-007ea4218a98" path="/var/lib/kubelet/pods/4e716825-de92-4c9d-9b36-007ea4218a98/volumes"
Oct 30 04:30:59.424294 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.416213    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p"]
Oct 30 04:30:59.424294 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.421112    4874 cpu_manager.go:414] "containerMap: removing container" podUID="4e716825-de92-4c9d-9b36-007ea4218a98" containerName="istio-proxy"
Oct 30 04:30:59.424294 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.421139    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e716825-de92-4c9d-9b36-007ea4218a98" containerName="istio-proxy"
Oct 30 04:30:59.424294 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.421314    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e716825-de92-4c9d-9b36-007ea4218a98" containerName="istio-proxy"
Oct 30 04:30:59.424294 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.421333    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e716825-de92-4c9d-9b36-007ea4218a98" containerName="istio-proxy"
Oct 30 04:30:59.434321 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.433409    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p"
Oct 30 04:30:59.447756 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.447723    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p"]
Oct 30 04:30:59.449257 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.449220    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-dbabed\""
Oct 30 04:30:59.450352 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.449497    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-dbabed-dockercfg-g4g92\""
Oct 30 04:30:59.550954 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.550919    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6624f77b-35f2-4d06-8a50-87acab5b9c14-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p\" (UID: \"6624f77b-35f2-4d06-8a50-87acab5b9c14\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p"
Oct 30 04:30:59.551127 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.550976    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkdjh\" (UniqueName: \"kubernetes.io/projected/6624f77b-35f2-4d06-8a50-87acab5b9c14-kube-api-access-pkdjh\") pod \"payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p\" (UID: \"6624f77b-35f2-4d06-8a50-87acab5b9c14\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p"
Oct 30 04:30:59.577670 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.577639    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz"]
Oct 30 04:30:59.578214 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.578199    4874 cpu_manager.go:414] "containerMap: removing container" podUID="4e716825-de92-4c9d-9b36-007ea4218a98" containerName="istio-proxy"
Oct 30 04:30:59.578283 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.578215    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e716825-de92-4c9d-9b36-007ea4218a98" containerName="istio-proxy"
Oct 30 04:30:59.581560 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.581545    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz"
Oct 30 04:30:59.602070 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.602046    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz"]
Oct 30 04:30:59.652361 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.652335    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6624f77b-35f2-4d06-8a50-87acab5b9c14-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p\" (UID: \"6624f77b-35f2-4d06-8a50-87acab5b9c14\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p"
Oct 30 04:30:59.652504 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.652377    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkdjh\" (UniqueName: \"kubernetes.io/projected/6624f77b-35f2-4d06-8a50-87acab5b9c14-kube-api-access-pkdjh\") pod \"payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p\" (UID: \"6624f77b-35f2-4d06-8a50-87acab5b9c14\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p"
Oct 30 04:30:59.652988 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.652967    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6624f77b-35f2-4d06-8a50-87acab5b9c14-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p\" (UID: \"6624f77b-35f2-4d06-8a50-87acab5b9c14\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p"
Oct 30 04:30:59.677164 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.677104    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkdjh\" (UniqueName: \"kubernetes.io/projected/6624f77b-35f2-4d06-8a50-87acab5b9c14-kube-api-access-pkdjh\") pod \"payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p\" (UID: \"6624f77b-35f2-4d06-8a50-87acab5b9c14\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p"
Oct 30 04:30:59.753815 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.753778    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f9442e-5e24-4a41-bf74-89ffc2cf240a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz\" (UID: \"85f9442e-5e24-4a41-bf74-89ffc2cf240a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz"
Oct 30 04:30:59.754033 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.754009    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdtr6\" (UniqueName: \"kubernetes.io/projected/85f9442e-5e24-4a41-bf74-89ffc2cf240a-kube-api-access-pdtr6\") pod \"payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz\" (UID: \"85f9442e-5e24-4a41-bf74-89ffc2cf240a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz"
Oct 30 04:30:59.768552 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.768533    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p"
Oct 30 04:30:59.854787 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.854743    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdtr6\" (UniqueName: \"kubernetes.io/projected/85f9442e-5e24-4a41-bf74-89ffc2cf240a-kube-api-access-pdtr6\") pod \"payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz\" (UID: \"85f9442e-5e24-4a41-bf74-89ffc2cf240a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz"
Oct 30 04:30:59.854926 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.854899    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f9442e-5e24-4a41-bf74-89ffc2cf240a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz\" (UID: \"85f9442e-5e24-4a41-bf74-89ffc2cf240a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz"
Oct 30 04:30:59.855944 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.855915    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f9442e-5e24-4a41-bf74-89ffc2cf240a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz\" (UID: \"85f9442e-5e24-4a41-bf74-89ffc2cf240a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz"
Oct 30 04:30:59.877672 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.877640    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdtr6\" (UniqueName: \"kubernetes.io/projected/85f9442e-5e24-4a41-bf74-89ffc2cf240a-kube-api-access-pdtr6\") pod \"payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz\" (UID: \"85f9442e-5e24-4a41-bf74-89ffc2cf240a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz"
Oct 30 04:30:59.894614 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:30:59.894596    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz"
Oct 30 04:31:00.070318 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:00.070291    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p"]
Oct 30 04:31:00.071927 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:31:00.071897    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6624f77b_35f2_4d06_8a50_87acab5b9c14.slice/crio-2b341a9e4671e760943b97174318cd1a51c7cda16d3f18e7c2b949360f3d0f40 WatchSource:0}: Error finding container 2b341a9e4671e760943b97174318cd1a51c7cda16d3f18e7c2b949360f3d0f40: Status 404 returned error can't find the container with id 2b341a9e4671e760943b97174318cd1a51c7cda16d3f18e7c2b949360f3d0f40
Oct 30 04:31:00.213079 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:00.213053    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz"]
Oct 30 04:31:00.215579 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:31:00.215555    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f9442e_5e24_4a41_bf74_89ffc2cf240a.slice/crio-56c7aff095bceae98899f1e545588ef18f4a5d387e5dbc310b1ff82898be4750 WatchSource:0}: Error finding container 56c7aff095bceae98899f1e545588ef18f4a5d387e5dbc310b1ff82898be4750: Status 404 returned error can't find the container with id 56c7aff095bceae98899f1e545588ef18f4a5d387e5dbc310b1ff82898be4750
Oct 30 04:31:00.583283 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:00.583256    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn"
Oct 30 04:31:00.614807 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:00.614771    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p" event={"ID":"6624f77b-35f2-4d06-8a50-87acab5b9c14","Type":"ContainerStarted","Data":"cf77fd36066308fa97e0a4257fea9294fc527d2c1e6b42124369501d54863486"}
Oct 30 04:31:00.614973 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:00.614828    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p" event={"ID":"6624f77b-35f2-4d06-8a50-87acab5b9c14","Type":"ContainerStarted","Data":"2b341a9e4671e760943b97174318cd1a51c7cda16d3f18e7c2b949360f3d0f40"}
Oct 30 04:31:00.614973 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:00.614937    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p"
Oct 30 04:31:00.616451 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:00.616422    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz" event={"ID":"85f9442e-5e24-4a41-bf74-89ffc2cf240a","Type":"ContainerStarted","Data":"e5144aa21f134d6290752e821471830c6772044c7a2dfd38017518007c17e6c0"}
Oct 30 04:31:00.616594 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:00.616476    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz" event={"ID":"85f9442e-5e24-4a41-bf74-89ffc2cf240a","Type":"ContainerStarted","Data":"56c7aff095bceae98899f1e545588ef18f4a5d387e5dbc310b1ff82898be4750"}
Oct 30 04:31:00.616594 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:00.616505    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz"
Oct 30 04:31:00.692906 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:00.692841    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz" podStartSLOduration=1.692826701 podStartE2EDuration="1.692826701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:31:00.691681336 +0000 UTC m=+1980.104076947" watchObservedRunningTime="2026-10-30 04:31:00.692826701 +0000 UTC m=+1980.105222274"
Oct 30 04:31:00.735641 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:00.735584    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p" podStartSLOduration=1.735567444 podStartE2EDuration="1.735567444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:31:00.729838265 +0000 UTC m=+1980.142233838" watchObservedRunningTime="2026-10-30 04:31:00.735567444 +0000 UTC m=+1980.147963022"
Oct 30 04:31:09.407122 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.407086    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"]
Oct 30 04:31:09.413297 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.413269    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.428134 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.428106    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-448a3a-openshift-default-dockercfg-vtlc5\""
Oct 30 04:31:09.443153 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.443123    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"]
Oct 30 04:31:09.498062 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.498027    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-data\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.498233 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.498066    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-token\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.498233 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.498100    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-workload-certs\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.498233 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.498131    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-envoy\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.498233 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.498164    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-podinfo\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.498233 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.498190    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt76g\" (UniqueName: \"kubernetes.io/projected/3f12d804-4aab-4672-af70-0f668cbb63a9-kube-api-access-kt76g\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.498233 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.498209    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-credential-socket\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.498454 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.498255    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f12d804-4aab-4672-af70-0f668cbb63a9-istiod-ca-cert\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.498454 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.498287    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-workload-socket\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.599034 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.598986    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-workload-certs\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.599226 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.599062    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-envoy\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.599226 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.599101    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-podinfo\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.599226 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.599120    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt76g\" (UniqueName: \"kubernetes.io/projected/3f12d804-4aab-4672-af70-0f668cbb63a9-kube-api-access-kt76g\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.599443 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.599265    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-credential-socket\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.599443 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.599322    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f12d804-4aab-4672-af70-0f668cbb63a9-istiod-ca-cert\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.599443 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.599374    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-workload-socket\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.599612 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.599442    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-workload-certs\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.599612 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.599522    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-data\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.599612 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.599556    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-token\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.599612 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.599567    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-credential-socket\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.599820 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.599749    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-workload-socket\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.599943 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.599916    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-data\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.600155 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.600131    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f12d804-4aab-4672-af70-0f668cbb63a9-istiod-ca-cert\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.601513 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.601492    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-envoy\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.601905 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.601885    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-podinfo\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.641217 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.641075    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt76g\" (UniqueName: \"kubernetes.io/projected/3f12d804-4aab-4672-af70-0f668cbb63a9-kube-api-access-kt76g\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.667322 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.664511    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-token\") pod \"e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") " pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:09.737262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:09.733368    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:10.121905 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:10.121864    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"]
Oct 30 04:31:10.123103 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:31:10.123075    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f12d804_4aab_4672_af70_0f668cbb63a9.slice/crio-8fa4332237630378eb36caa7b670ad0c97c998121da4f2db12200bb5d5299e2d WatchSource:0}: Error finding container 8fa4332237630378eb36caa7b670ad0c97c998121da4f2db12200bb5d5299e2d: Status 404 returned error can't find the container with id 8fa4332237630378eb36caa7b670ad0c97c998121da4f2db12200bb5d5299e2d
Oct 30 04:31:10.125921 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:10.125878    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:31:10.126003 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:10.125967    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:31:10.126041 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:10.126008    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:31:10.283265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:10.274192    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:31:10.829017 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:10.828976    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx" event={"ID":"3f12d804-4aab-4672-af70-0f668cbb63a9","Type":"ContainerStarted","Data":"bc48517e00eb1827516ad79a3321d8299a415c407134f7128bd98fcc68a7c69f"}
Oct 30 04:31:10.829431 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:10.829028    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx" event={"ID":"3f12d804-4aab-4672-af70-0f668cbb63a9","Type":"ContainerStarted","Data":"8fa4332237630378eb36caa7b670ad0c97c998121da4f2db12200bb5d5299e2d"}
Oct 30 04:31:10.897818 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:10.897747    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx" podStartSLOduration=1.8977273449999998 podStartE2EDuration="1.897727345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:31:10.887442347 +0000 UTC m=+1990.299837921" watchObservedRunningTime="2026-10-30 04:31:10.897727345 +0000 UTC m=+1990.310122918"
Oct 30 04:31:11.625435 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:11.625402    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz"
Oct 30 04:31:11.625634 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:11.625470    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p"
Oct 30 04:31:11.739403 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:11.739364    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:11.744996 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:11.744968    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:11.836998 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:11.836190    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:11.837508 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:11.837367    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:31:16.320366 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:16.320332    4874 patch_prober.go:29] interesting pod/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 04:31:16.320757 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:16.320391    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp" podUID="4ded57b2-1fa2-414f-9f3b-b598eee59dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 04:31:17.874956 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:17.874914    4874 generic.go:356] "Generic (PLEG): container finished" podID="4ded57b2-1fa2-414f-9f3b-b598eee59dad" containerID="04e566564f84cf9972937d9e23690af8b7698c68c76bb00f9d86c254a07bef16" exitCode=0
Oct 30 04:31:17.875515 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:17.874989    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp" event={"ID":"4ded57b2-1fa2-414f-9f3b-b598eee59dad","Type":"ContainerDied","Data":"04e566564f84cf9972937d9e23690af8b7698c68c76bb00f9d86c254a07bef16"}
Oct 30 04:31:17.875594 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:17.875526    4874 scope.go:122] "RemoveContainer" containerID="04e566564f84cf9972937d9e23690af8b7698c68c76bb00f9d86c254a07bef16"
Oct 30 04:31:17.877450 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:17.877417    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:31:17.877533 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:17.877513    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:31:17.877572 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:17.877561    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:31:18.223539 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.223460    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"]
Oct 30 04:31:18.227719 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.227692    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:31:18.241719 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.241690    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f7a36d\"/\"default-dockercfg-4vhdj\""
Oct 30 04:31:18.244769 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.244747    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f7a36d\"/\"kube-root-ca.crt\""
Oct 30 04:31:18.244903 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.244769    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f7a36d\"/\"openshift-service-ca.crt\""
Oct 30 04:31:18.248557 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.248532    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"]
Oct 30 04:31:18.320014 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.319980    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f7a36d\"/\"ipp-route-f7a36d-kserve-self-signed-certs\""
Oct 30 04:31:18.355923 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.355892    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-model-cache\") pod \"ipp-route-f7a36d-kserve-65bd4f59fc-25xn4\" (UID: \"095312cb-be01-4f88-8cc0-87eecca32f5d\") " pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:31:18.356104 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.356067    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095312cb-be01-4f88-8cc0-87eecca32f5d-tls-certs\") pod \"ipp-route-f7a36d-kserve-65bd4f59fc-25xn4\" (UID: \"095312cb-be01-4f88-8cc0-87eecca32f5d\") " pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:31:18.356104 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.356098    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-tmp-dir\") pod \"ipp-route-f7a36d-kserve-65bd4f59fc-25xn4\" (UID: \"095312cb-be01-4f88-8cc0-87eecca32f5d\") " pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:31:18.356225 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.356137    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-home\") pod \"ipp-route-f7a36d-kserve-65bd4f59fc-25xn4\" (UID: \"095312cb-be01-4f88-8cc0-87eecca32f5d\") " pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:31:18.356225 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.356190    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-dshm\") pod \"ipp-route-f7a36d-kserve-65bd4f59fc-25xn4\" (UID: \"095312cb-be01-4f88-8cc0-87eecca32f5d\") " pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:31:18.356390 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.356360    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq8q6\" (UniqueName: \"kubernetes.io/projected/095312cb-be01-4f88-8cc0-87eecca32f5d-kube-api-access-gq8q6\") pod \"ipp-route-f7a36d-kserve-65bd4f59fc-25xn4\" (UID: \"095312cb-be01-4f88-8cc0-87eecca32f5d\") " pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:31:18.473299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.464371    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq8q6\" (UniqueName: \"kubernetes.io/projected/095312cb-be01-4f88-8cc0-87eecca32f5d-kube-api-access-gq8q6\") pod \"ipp-route-f7a36d-kserve-65bd4f59fc-25xn4\" (UID: \"095312cb-be01-4f88-8cc0-87eecca32f5d\") " pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:31:18.473299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.464498    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-model-cache\") pod \"ipp-route-f7a36d-kserve-65bd4f59fc-25xn4\" (UID: \"095312cb-be01-4f88-8cc0-87eecca32f5d\") " pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:31:18.473299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.464639    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095312cb-be01-4f88-8cc0-87eecca32f5d-tls-certs\") pod \"ipp-route-f7a36d-kserve-65bd4f59fc-25xn4\" (UID: \"095312cb-be01-4f88-8cc0-87eecca32f5d\") " pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:31:18.473299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.464680    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-tmp-dir\") pod \"ipp-route-f7a36d-kserve-65bd4f59fc-25xn4\" (UID: \"095312cb-be01-4f88-8cc0-87eecca32f5d\") " pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:31:18.473299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.464727    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-home\") pod \"ipp-route-f7a36d-kserve-65bd4f59fc-25xn4\" (UID: \"095312cb-be01-4f88-8cc0-87eecca32f5d\") " pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:31:18.473299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.464751    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-dshm\") pod \"ipp-route-f7a36d-kserve-65bd4f59fc-25xn4\" (UID: \"095312cb-be01-4f88-8cc0-87eecca32f5d\") " pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:31:18.473299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.465690    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-model-cache\") pod \"ipp-route-f7a36d-kserve-65bd4f59fc-25xn4\" (UID: \"095312cb-be01-4f88-8cc0-87eecca32f5d\") " pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:31:18.473299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.466381    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-tmp-dir\") pod \"ipp-route-f7a36d-kserve-65bd4f59fc-25xn4\" (UID: \"095312cb-be01-4f88-8cc0-87eecca32f5d\") " pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:31:18.473299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.467116    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-home\") pod \"ipp-route-f7a36d-kserve-65bd4f59fc-25xn4\" (UID: \"095312cb-be01-4f88-8cc0-87eecca32f5d\") " pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:31:18.484355 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.479441    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-dshm\") pod \"ipp-route-f7a36d-kserve-65bd4f59fc-25xn4\" (UID: \"095312cb-be01-4f88-8cc0-87eecca32f5d\") " pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:31:18.484355 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.482046    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095312cb-be01-4f88-8cc0-87eecca32f5d-tls-certs\") pod \"ipp-route-f7a36d-kserve-65bd4f59fc-25xn4\" (UID: \"095312cb-be01-4f88-8cc0-87eecca32f5d\") " pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:31:18.502869 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.499272    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq8q6\" (UniqueName: \"kubernetes.io/projected/095312cb-be01-4f88-8cc0-87eecca32f5d-kube-api-access-gq8q6\") pod \"ipp-route-f7a36d-kserve-65bd4f59fc-25xn4\" (UID: \"095312cb-be01-4f88-8cc0-87eecca32f5d\") " pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:31:18.549647 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.548506    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:31:18.864062 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.864038    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"]
Oct 30 04:31:18.891850 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.891558    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp" event={"ID":"4ded57b2-1fa2-414f-9f3b-b598eee59dad","Type":"ContainerStarted","Data":"f2462aa95d441531f3f480b76676f133e50032e57234e0f5f8183ab6a807ee2e"}
Oct 30 04:31:18.893665 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:18.893558    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4" event={"ID":"095312cb-be01-4f88-8cc0-87eecca32f5d","Type":"ContainerStarted","Data":"83daf016d9aff4ff35daad712412cbd364952914a582360786070cac93e6f024"}
Oct 30 04:31:19.076267 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:19.069618    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:31:19.076267 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:19.069738    4874 patch_prober.go:29] interesting pod/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 04:31:19.076267 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:19.069779    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp" podUID="4ded57b2-1fa2-414f-9f3b-b598eee59dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 04:31:19.948628 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:19.920009    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4" event={"ID":"095312cb-be01-4f88-8cc0-87eecca32f5d","Type":"ContainerStarted","Data":"bf06354d826110001d5dd86841f10bbb45449eb1c44e15eaa85ce9f209a6beb2"}
Oct 30 04:31:19.960407 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:19.960086    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4" podStartSLOduration=1.9600671570000001 podStartE2EDuration="1.960067157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:31:19.959171686 +0000 UTC m=+1999.371567258" watchObservedRunningTime="2026-10-30 04:31:19.960067157 +0000 UTC m=+1999.372462731"
Oct 30 04:31:20.087262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:20.074633    4874 patch_prober.go:29] interesting pod/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 04:31:20.087262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:20.074695    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp" podUID="4ded57b2-1fa2-414f-9f3b-b598eee59dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 04:31:20.549485 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:20.549452    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:31:21.067634 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:21.067598    4874 patch_prober.go:29] interesting pod/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 04:31:21.068204 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:21.068174    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp" podUID="4ded57b2-1fa2-414f-9f3b-b598eee59dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 04:31:21.932627 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:21.932587    4874 generic.go:356] "Generic (PLEG): container finished" podID="1f0cc4fd-bf2b-4642-ab23-e6649237f0fb" containerID="f640d6169b90ad661207d8685aba0f6f09fdfe54c9f5e0896727909458157434" exitCode=0
Oct 30 04:31:21.932833 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:21.932659    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v" event={"ID":"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb","Type":"ContainerDied","Data":"f640d6169b90ad661207d8685aba0f6f09fdfe54c9f5e0896727909458157434"}
Oct 30 04:31:21.933202 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:21.933184    4874 scope.go:122] "RemoveContainer" containerID="f640d6169b90ad661207d8685aba0f6f09fdfe54c9f5e0896727909458157434"
Oct 30 04:31:21.934863 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:21.934838    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:31:21.934931 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:21.934923    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:31:21.934975 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:21.934964    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:31:22.091260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:22.090372    4874 patch_prober.go:29] interesting pod/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 04:31:22.091260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:22.090429    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp" podUID="4ded57b2-1fa2-414f-9f3b-b598eee59dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 04:31:22.941060 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:22.941026    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v" event={"ID":"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb","Type":"ContainerStarted","Data":"ebf13b275d089f49ed0872ebffe0c7c84e05f58637835d6643cb1b52a35234d5"}
Oct 30 04:31:23.078264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:23.076610    4874 patch_prober.go:29] interesting pod/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 04:31:23.078264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:23.076674    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp" podUID="4ded57b2-1fa2-414f-9f3b-b598eee59dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 04:31:23.717339 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:23.717305    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:31:23.718228 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:23.717718    4874 patch_prober.go:29] interesting pod/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 04:31:23.718228 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:23.717981    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v" podUID="1f0cc4fd-bf2b-4642-ab23-e6649237f0fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 04:31:24.123268 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:24.122829    4874 patch_prober.go:29] interesting pod/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 04:31:24.123268 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:24.122885    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp" podUID="4ded57b2-1fa2-414f-9f3b-b598eee59dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 04:31:24.716689 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:24.716647    4874 patch_prober.go:29] interesting pod/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 04:31:24.716932 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:24.716704    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v" podUID="1f0cc4fd-bf2b-4642-ab23-e6649237f0fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 04:31:25.084536 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:25.084502    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:31:25.089271 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:25.086455    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:31:25.090297 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:25.090104    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:31:25.093105 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:25.093077    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"]
Oct 30 04:31:25.093318 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:25.093294    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6" podUID="37fde430-0d9b-4c2e-b4d2-97e36fa58004" containerName="istio-proxy" containerID="cri-o://1a339d105362360e7fba03f9c60894b76b296fe77359cfe2cadbc15dbc6abe58" gracePeriod=30
Oct 30 04:31:25.614033 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:31:25.614000    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:31:25.716428 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:25.716390    4874 patch_prober.go:29] interesting pod/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 04:31:25.716607 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:25.716449    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v" podUID="1f0cc4fd-bf2b-4642-ab23-e6649237f0fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 04:31:26.716698 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:26.716656    4874 patch_prober.go:29] interesting pod/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 04:31:26.717109 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:26.716717    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v" podUID="1f0cc4fd-bf2b-4642-ab23-e6649237f0fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 04:31:27.360017 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:27.359982    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:31:27.716367 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:27.716271    4874 patch_prober.go:29] interesting pod/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 04:31:27.716367 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:27.716339    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v" podUID="1f0cc4fd-bf2b-4642-ab23-e6649237f0fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 04:31:28.550043 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:28.550008    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:31:28.564653 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:28.564627    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:31:28.720657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:28.720627    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:31:28.736725 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:28.736698    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:31:29.017682 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:29.017651    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:31:30.360211 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.360156    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:31:30.451720 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.451686    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-workload-socket\") pod \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") "
Oct 30 04:31:30.451879 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.451746    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-data\") pod \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") "
Oct 30 04:31:30.451879 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.451774    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-envoy\") pod \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") "
Oct 30 04:31:30.451879 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.451808    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istiod-ca-cert\") pod \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") "
Oct 30 04:31:30.451879 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.451844    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37fde430-0d9b-4c2e-b4d2-97e36fa58004-kube-api-access-v8247\" (UniqueName: \"kubernetes.io/projected/37fde430-0d9b-4c2e-b4d2-97e36fa58004-kube-api-access-v8247\") pod \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") "
Oct 30 04:31:30.452102 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.451894    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-podinfo\") pod \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") "
Oct 30 04:31:30.452102 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.452006    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-workload-socket" pod "37fde430-0d9b-4c2e-b4d2-97e36fa58004" (UID: "37fde430-0d9b-4c2e-b4d2-97e36fa58004"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:31:30.452102 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.452041    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-token\" (UniqueName: \"kubernetes.io/projected/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-token\") pod \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") "
Oct 30 04:31:30.452281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.452130    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-data" pod "37fde430-0d9b-4c2e-b4d2-97e36fa58004" (UID: "37fde430-0d9b-4c2e-b4d2-97e36fa58004"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:31:30.452281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.452156    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-workload-certs\") pod \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") "
Oct 30 04:31:30.452281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.452269    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-credential-socket\") pod \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\" (UID: \"37fde430-0d9b-4c2e-b4d2-97e36fa58004\") "
Oct 30 04:31:30.452461 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.452428    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-workload-certs" pod "37fde430-0d9b-4c2e-b4d2-97e36fa58004" (UID: "37fde430-0d9b-4c2e-b4d2-97e36fa58004"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:31:30.452519 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.452450    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istiod-ca-cert" pod "37fde430-0d9b-4c2e-b4d2-97e36fa58004" (UID: "37fde430-0d9b-4c2e-b4d2-97e36fa58004"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:31:30.452678 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.452656    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-credential-socket" pod "37fde430-0d9b-4c2e-b4d2-97e36fa58004" (UID: "37fde430-0d9b-4c2e-b4d2-97e36fa58004"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:31:30.452801 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.452782    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:31:30.452863 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.452808    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:31:30.452863 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.452824    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:31:30.452863 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.452838    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:31:30.452863 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.452853    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:31:30.454683 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.454634    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-envoy" pod "37fde430-0d9b-4c2e-b4d2-97e36fa58004" (UID: "37fde430-0d9b-4c2e-b4d2-97e36fa58004"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:31:30.454912 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.454894    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37fde430-0d9b-4c2e-b4d2-97e36fa58004-kube-api-access-v8247" pod "37fde430-0d9b-4c2e-b4d2-97e36fa58004" (UID: "37fde430-0d9b-4c2e-b4d2-97e36fa58004"). InnerVolumeSpecName "kube-api-access-v8247". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:31:30.455073 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.455048    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-podinfo" pod "37fde430-0d9b-4c2e-b4d2-97e36fa58004" (UID: "37fde430-0d9b-4c2e-b4d2-97e36fa58004"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:31:30.455312 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.455287    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-token" pod "37fde430-0d9b-4c2e-b4d2-97e36fa58004" (UID: "37fde430-0d9b-4c2e-b4d2-97e36fa58004"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:31:30.553737 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.553705    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:31:30.553737 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.553735    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8247\" (UniqueName: \"kubernetes.io/projected/37fde430-0d9b-4c2e-b4d2-97e36fa58004-kube-api-access-v8247\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:31:30.553737 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.553745    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:31:30.553960 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:30.553754    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fde430-0d9b-4c2e-b4d2-97e36fa58004-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:31:31.020531 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:31.020448    4874 generic.go:356] "Generic (PLEG): container finished" podID="37fde430-0d9b-4c2e-b4d2-97e36fa58004" containerID="1a339d105362360e7fba03f9c60894b76b296fe77359cfe2cadbc15dbc6abe58" exitCode=0
Oct 30 04:31:31.020531 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:31.020489    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"
Oct 30 04:31:31.020696 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:31.020538    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6" event={"ID":"37fde430-0d9b-4c2e-b4d2-97e36fa58004","Type":"ContainerDied","Data":"1a339d105362360e7fba03f9c60894b76b296fe77359cfe2cadbc15dbc6abe58"}
Oct 30 04:31:31.020696 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:31.020591    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6" event={"ID":"37fde430-0d9b-4c2e-b4d2-97e36fa58004","Type":"ContainerDied","Data":"597ba0f3062b35b284562e4a6d75278c121c6b298a1e5a216d438fb0f20f2d32"}
Oct 30 04:31:31.020696 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:31.020618    4874 scope.go:122] "RemoveContainer" containerID="1a339d105362360e7fba03f9c60894b76b296fe77359cfe2cadbc15dbc6abe58"
Oct 30 04:31:31.030036 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:31.030015    4874 scope.go:122] "RemoveContainer" containerID="1a339d105362360e7fba03f9c60894b76b296fe77359cfe2cadbc15dbc6abe58"
Oct 30 04:31:31.030329 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:31:31.030308    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a339d105362360e7fba03f9c60894b76b296fe77359cfe2cadbc15dbc6abe58\": container with ID starting with 1a339d105362360e7fba03f9c60894b76b296fe77359cfe2cadbc15dbc6abe58 not found: ID does not exist" containerID="1a339d105362360e7fba03f9c60894b76b296fe77359cfe2cadbc15dbc6abe58"
Oct 30 04:31:31.030396 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:31.030337    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a339d105362360e7fba03f9c60894b76b296fe77359cfe2cadbc15dbc6abe58"} err="failed to get container status \"1a339d105362360e7fba03f9c60894b76b296fe77359cfe2cadbc15dbc6abe58\": rpc error: code = NotFound desc = could not find container \"1a339d105362360e7fba03f9c60894b76b296fe77359cfe2cadbc15dbc6abe58\": container with ID starting with 1a339d105362360e7fba03f9c60894b76b296fe77359cfe2cadbc15dbc6abe58 not found: ID does not exist"
Oct 30 04:31:31.081701 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:31.081674    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"]
Oct 30 04:31:31.093229 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:31.093205    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-zmjw6"]
Oct 30 04:31:32.729825 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:32.729791    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37fde430-0d9b-4c2e-b4d2-97e36fa58004" path="/var/lib/kubelet/pods/37fde430-0d9b-4c2e-b4d2-97e36fa58004/volumes"
Oct 30 04:31:40.148882 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:40.148847    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd"]
Oct 30 04:31:40.149494 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:40.149475    4874 cpu_manager.go:414] "containerMap: removing container" podUID="37fde430-0d9b-4c2e-b4d2-97e36fa58004" containerName="istio-proxy"
Oct 30 04:31:40.149562 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:40.149498    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37fde430-0d9b-4c2e-b4d2-97e36fa58004" containerName="istio-proxy"
Oct 30 04:31:40.149631 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:40.149620    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="37fde430-0d9b-4c2e-b4d2-97e36fa58004" containerName="istio-proxy"
Oct 30 04:31:40.153232 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:40.153213    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd"
Oct 30 04:31:40.181427 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:40.181396    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-448a3a-serving-cert\""
Oct 30 04:31:40.184599 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:40.184575    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd"]
Oct 30 04:31:40.262484 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:40.262455    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a534d3-0927-4aa0-882a-c5734776b387-metrics-certs\") pod \"maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd\" (UID: \"80a534d3-0927-4aa0-882a-c5734776b387\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd"
Oct 30 04:31:40.262722 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:40.262625    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a534d3-0927-4aa0-882a-c5734776b387-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd\" (UID: \"80a534d3-0927-4aa0-882a-c5734776b387\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd"
Oct 30 04:31:40.262722 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:40.262660    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80a534d3-0927-4aa0-882a-c5734776b387-maas-api-tls\") pod \"maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd\" (UID: \"80a534d3-0927-4aa0-882a-c5734776b387\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd"
Oct 30 04:31:40.262722 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:40.262695    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92p9g\" (UniqueName: \"kubernetes.io/projected/80a534d3-0927-4aa0-882a-c5734776b387-kube-api-access-92p9g\") pod \"maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd\" (UID: \"80a534d3-0927-4aa0-882a-c5734776b387\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd"
Oct 30 04:31:40.381263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:40.373104    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a534d3-0927-4aa0-882a-c5734776b387-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd\" (UID: \"80a534d3-0927-4aa0-882a-c5734776b387\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd"
Oct 30 04:31:40.381263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:40.373169    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80a534d3-0927-4aa0-882a-c5734776b387-maas-api-tls\") pod \"maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd\" (UID: \"80a534d3-0927-4aa0-882a-c5734776b387\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd"
Oct 30 04:31:40.381263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:40.373221    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92p9g\" (UniqueName: \"kubernetes.io/projected/80a534d3-0927-4aa0-882a-c5734776b387-kube-api-access-92p9g\") pod \"maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd\" (UID: \"80a534d3-0927-4aa0-882a-c5734776b387\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd"
Oct 30 04:31:40.381263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:40.374472    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a534d3-0927-4aa0-882a-c5734776b387-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd\" (UID: \"80a534d3-0927-4aa0-882a-c5734776b387\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd"
Oct 30 04:31:40.381263 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:31:40.374591    4874 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-serving-cert: secret "maas-api-e2e-praxis-b-448a3a-serving-cert" not found
Oct 30 04:31:40.381263 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:31:40.375108    4874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80a534d3-0927-4aa0-882a-c5734776b387-maas-api-tls podName:80a534d3-0927-4aa0-882a-c5734776b387 nodeName:}" failed. No retries permitted until 2026-10-30 04:31:40.87504278 +0000 UTC m=+2020.287438330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/80a534d3-0927-4aa0-882a-c5734776b387-maas-api-tls") pod "maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd" (UID: "80a534d3-0927-4aa0-882a-c5734776b387") : secret "maas-api-e2e-praxis-b-448a3a-serving-cert" not found
Oct 30 04:31:40.381263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:40.375224    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a534d3-0927-4aa0-882a-c5734776b387-metrics-certs\") pod \"maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd\" (UID: \"80a534d3-0927-4aa0-882a-c5734776b387\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd"
Oct 30 04:31:40.383461 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:40.383402    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a534d3-0927-4aa0-882a-c5734776b387-metrics-certs\") pod \"maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd\" (UID: \"80a534d3-0927-4aa0-882a-c5734776b387\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd"
Oct 30 04:31:40.399591 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:40.399526    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92p9g\" (UniqueName: \"kubernetes.io/projected/80a534d3-0927-4aa0-882a-c5734776b387-kube-api-access-92p9g\") pod \"maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd\" (UID: \"80a534d3-0927-4aa0-882a-c5734776b387\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd"
Oct 30 04:31:40.880209 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:40.880179    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80a534d3-0927-4aa0-882a-c5734776b387-maas-api-tls\") pod \"maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd\" (UID: \"80a534d3-0927-4aa0-882a-c5734776b387\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd"
Oct 30 04:31:40.882657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:40.882632    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80a534d3-0927-4aa0-882a-c5734776b387-maas-api-tls\") pod \"maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd\" (UID: \"80a534d3-0927-4aa0-882a-c5734776b387\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd"
Oct 30 04:31:41.066020 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:41.065989    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd"
Oct 30 04:31:41.389999 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:41.389976    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd"]
Oct 30 04:31:41.392824 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:31:41.392795    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a534d3_0927_4aa0_882a_c5734776b387.slice/crio-f33a065d4c4c8e717c73ede3093eaf9a4b759cb1fc92732ccaa003fdb47df2ca WatchSource:0}: Error finding container f33a065d4c4c8e717c73ede3093eaf9a4b759cb1fc92732ccaa003fdb47df2ca: Status 404 returned error can't find the container with id f33a065d4c4c8e717c73ede3093eaf9a4b759cb1fc92732ccaa003fdb47df2ca
Oct 30 04:31:42.110676 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:42.110631    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd" event={"ID":"80a534d3-0927-4aa0-882a-c5734776b387","Type":"ContainerStarted","Data":"020e19d403e5f4148d0bb8e8fa23f12aa62fe6f3e925c17807e5fbd0dcd2d064"}
Oct 30 04:31:42.110884 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:42.110726    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd" event={"ID":"80a534d3-0927-4aa0-882a-c5734776b387","Type":"ContainerStarted","Data":"f33a065d4c4c8e717c73ede3093eaf9a4b759cb1fc92732ccaa003fdb47df2ca"}
Oct 30 04:31:42.110884 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:42.110756    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd"
Oct 30 04:31:42.130205 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:42.130157    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:31:42.159515 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:42.159462    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd" podStartSLOduration=1.854555443 podStartE2EDuration="2.159449407s" totalImagesPullingTime="304.893964ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:31:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:31:42.157021106 +0000 UTC m=+2021.569416678" watchObservedRunningTime="2026-10-30 04:31:42.159449407 +0000 UTC m=+2021.571844980"
Oct 30 04:31:48.124301 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:48.124269    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd"
Oct 30 04:31:54.437571 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:54.437537    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n"]
Oct 30 04:31:55.608849 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:31:55.608812    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:31:59.438607 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:59.438556    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n" podUID="1d4d2573-179f-46bc-be32-e3dcc140437f" containerName="payload-pre-processing" containerID="cri-o://b9a4be4ff309eb77a20012241822eb52a263c5190148c877ae6324fa3dcd9a92" gracePeriod=25
Oct 30 04:31:59.749233 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:59.749203    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n"
Oct 30 04:31:59.860196 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:59.860164    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d4d2573-179f-46bc-be32-e3dcc140437f-kube-api-access-9prc6\" (UniqueName: \"kubernetes.io/projected/1d4d2573-179f-46bc-be32-e3dcc140437f-kube-api-access-9prc6\") pod \"1d4d2573-179f-46bc-be32-e3dcc140437f\" (UID: \"1d4d2573-179f-46bc-be32-e3dcc140437f\") "
Oct 30 04:31:59.860398 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:59.860254    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d4d2573-179f-46bc-be32-e3dcc140437f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d4d2573-179f-46bc-be32-e3dcc140437f-plugins-config-volume\") pod \"1d4d2573-179f-46bc-be32-e3dcc140437f\" (UID: \"1d4d2573-179f-46bc-be32-e3dcc140437f\") "
Oct 30 04:31:59.860668 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:59.860643    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d4d2573-179f-46bc-be32-e3dcc140437f-plugins-config-volume" pod "1d4d2573-179f-46bc-be32-e3dcc140437f" (UID: "1d4d2573-179f-46bc-be32-e3dcc140437f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:31:59.860782 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:59.860763    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d4d2573-179f-46bc-be32-e3dcc140437f-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:31:59.862433 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:59.862409    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d4d2573-179f-46bc-be32-e3dcc140437f-kube-api-access-9prc6" pod "1d4d2573-179f-46bc-be32-e3dcc140437f" (UID: "1d4d2573-179f-46bc-be32-e3dcc140437f"). InnerVolumeSpecName "kube-api-access-9prc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:31:59.961842 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:31:59.961806    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9prc6\" (UniqueName: \"kubernetes.io/projected/1d4d2573-179f-46bc-be32-e3dcc140437f-kube-api-access-9prc6\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:32:00.236123 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:00.236085    4874 generic.go:356] "Generic (PLEG): container finished" podID="1d4d2573-179f-46bc-be32-e3dcc140437f" containerID="b9a4be4ff309eb77a20012241822eb52a263c5190148c877ae6324fa3dcd9a92" exitCode=0
Oct 30 04:32:00.236336 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:00.236137    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n"
Oct 30 04:32:00.236336 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:00.236168    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n" event={"ID":"1d4d2573-179f-46bc-be32-e3dcc140437f","Type":"ContainerDied","Data":"b9a4be4ff309eb77a20012241822eb52a263c5190148c877ae6324fa3dcd9a92"}
Oct 30 04:32:00.236336 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:00.236227    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n" event={"ID":"1d4d2573-179f-46bc-be32-e3dcc140437f","Type":"ContainerDied","Data":"a235dd90cbceee370b8038f5b7b2111cbcd622068779f4c5d7c8886badc34082"}
Oct 30 04:32:00.236336 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:00.236282    4874 scope.go:122] "RemoveContainer" containerID="b9a4be4ff309eb77a20012241822eb52a263c5190148c877ae6324fa3dcd9a92"
Oct 30 04:32:00.246868 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:00.246850    4874 scope.go:122] "RemoveContainer" containerID="b9a4be4ff309eb77a20012241822eb52a263c5190148c877ae6324fa3dcd9a92"
Oct 30 04:32:00.247135 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:32:00.247114    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a4be4ff309eb77a20012241822eb52a263c5190148c877ae6324fa3dcd9a92\": container with ID starting with b9a4be4ff309eb77a20012241822eb52a263c5190148c877ae6324fa3dcd9a92 not found: ID does not exist" containerID="b9a4be4ff309eb77a20012241822eb52a263c5190148c877ae6324fa3dcd9a92"
Oct 30 04:32:00.247199 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:00.247144    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a4be4ff309eb77a20012241822eb52a263c5190148c877ae6324fa3dcd9a92"} err="failed to get container status \"b9a4be4ff309eb77a20012241822eb52a263c5190148c877ae6324fa3dcd9a92\": rpc error: code = NotFound desc = could not find container \"b9a4be4ff309eb77a20012241822eb52a263c5190148c877ae6324fa3dcd9a92\": container with ID starting with b9a4be4ff309eb77a20012241822eb52a263c5190148c877ae6324fa3dcd9a92 not found: ID does not exist"
Oct 30 04:32:00.310024 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:00.309991    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n"]
Oct 30 04:32:00.321512 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:00.321478    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-58cdc8-8c8857759-sd87n"]
Oct 30 04:32:00.732554 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:00.732521    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d4d2573-179f-46bc-be32-e3dcc140437f" path="/var/lib/kubelet/pods/1d4d2573-179f-46bc-be32-e3dcc140437f/volumes"
Oct 30 04:32:01.392559 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:01.392516    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-71b161-668476b448-8489m"]
Oct 30 04:32:01.423512 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:01.423478    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr"]
Oct 30 04:32:06.393226 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:06.393181    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-71b161-668476b448-8489m" podUID="96a65bb0-e452-4068-ba0e-7145782c3b1f" containerName="payload-processing" containerID="cri-o://323935c9e1f07702be8b088c545084688c58b4c4830f737504790294c5b0fe7b" gracePeriod=25
Oct 30 04:32:06.424109 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:06.424061    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr" podUID="71b394ae-aff0-45fb-947e-05cd7bbd436a" containerName="payload-pre-processing" containerID="cri-o://ff8d8d0d40bba0e262adb0d9fb64f1b902805332711d8162aef8733d591d7cb9" gracePeriod=25
Oct 30 04:32:06.725339 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:06.725308    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-71b161-668476b448-8489m"
Oct 30 04:32:06.780587 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:06.780550    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr"
Oct 30 04:32:06.809291 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:06.809259    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a65bb0-e452-4068-ba0e-7145782c3b1f-kube-api-access-qq2sq\" (UniqueName: \"kubernetes.io/projected/96a65bb0-e452-4068-ba0e-7145782c3b1f-kube-api-access-qq2sq\") pod \"96a65bb0-e452-4068-ba0e-7145782c3b1f\" (UID: \"96a65bb0-e452-4068-ba0e-7145782c3b1f\") "
Oct 30 04:32:06.809513 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:06.809493    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96a65bb0-e452-4068-ba0e-7145782c3b1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a65bb0-e452-4068-ba0e-7145782c3b1f-plugins-config-volume\") pod \"96a65bb0-e452-4068-ba0e-7145782c3b1f\" (UID: \"96a65bb0-e452-4068-ba0e-7145782c3b1f\") "
Oct 30 04:32:06.809844 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:06.809816    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a65bb0-e452-4068-ba0e-7145782c3b1f-plugins-config-volume" pod "96a65bb0-e452-4068-ba0e-7145782c3b1f" (UID: "96a65bb0-e452-4068-ba0e-7145782c3b1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:32:06.810958 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:06.810935    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a65bb0-e452-4068-ba0e-7145782c3b1f-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:32:06.811442 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:06.811421    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a65bb0-e452-4068-ba0e-7145782c3b1f-kube-api-access-qq2sq" pod "96a65bb0-e452-4068-ba0e-7145782c3b1f" (UID: "96a65bb0-e452-4068-ba0e-7145782c3b1f"). InnerVolumeSpecName "kube-api-access-qq2sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:32:06.912387 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:06.912351    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b394ae-aff0-45fb-947e-05cd7bbd436a-kube-api-access-76rdv\" (UniqueName: \"kubernetes.io/projected/71b394ae-aff0-45fb-947e-05cd7bbd436a-kube-api-access-76rdv\") pod \"71b394ae-aff0-45fb-947e-05cd7bbd436a\" (UID: \"71b394ae-aff0-45fb-947e-05cd7bbd436a\") "
Oct 30 04:32:06.912561 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:06.912448    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71b394ae-aff0-45fb-947e-05cd7bbd436a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b394ae-aff0-45fb-947e-05cd7bbd436a-plugins-config-volume\") pod \"71b394ae-aff0-45fb-947e-05cd7bbd436a\" (UID: \"71b394ae-aff0-45fb-947e-05cd7bbd436a\") "
Oct 30 04:32:06.912787 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:06.912769    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq2sq\" (UniqueName: \"kubernetes.io/projected/96a65bb0-e452-4068-ba0e-7145782c3b1f-kube-api-access-qq2sq\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:32:06.912880 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:06.912786    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b394ae-aff0-45fb-947e-05cd7bbd436a-plugins-config-volume" pod "71b394ae-aff0-45fb-947e-05cd7bbd436a" (UID: "71b394ae-aff0-45fb-947e-05cd7bbd436a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:32:06.914412 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:06.914392    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b394ae-aff0-45fb-947e-05cd7bbd436a-kube-api-access-76rdv" pod "71b394ae-aff0-45fb-947e-05cd7bbd436a" (UID: "71b394ae-aff0-45fb-947e-05cd7bbd436a"). InnerVolumeSpecName "kube-api-access-76rdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:32:07.014528 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:07.014497    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76rdv\" (UniqueName: \"kubernetes.io/projected/71b394ae-aff0-45fb-947e-05cd7bbd436a-kube-api-access-76rdv\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:32:07.014528 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:07.014530    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b394ae-aff0-45fb-947e-05cd7bbd436a-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:32:07.368456 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:07.368373    4874 generic.go:356] "Generic (PLEG): container finished" podID="96a65bb0-e452-4068-ba0e-7145782c3b1f" containerID="323935c9e1f07702be8b088c545084688c58b4c4830f737504790294c5b0fe7b" exitCode=0
Oct 30 04:32:07.368456 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:07.368421    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-71b161-668476b448-8489m"
Oct 30 04:32:07.368664 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:07.368462    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-71b161-668476b448-8489m" event={"ID":"96a65bb0-e452-4068-ba0e-7145782c3b1f","Type":"ContainerDied","Data":"323935c9e1f07702be8b088c545084688c58b4c4830f737504790294c5b0fe7b"}
Oct 30 04:32:07.368664 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:07.368513    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-71b161-668476b448-8489m" event={"ID":"96a65bb0-e452-4068-ba0e-7145782c3b1f","Type":"ContainerDied","Data":"685c302c1408926e017ae0bb96c57bf8456cd2bfaa57836ac6ca6458a4fff0e3"}
Oct 30 04:32:07.368664 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:07.368536    4874 scope.go:122] "RemoveContainer" containerID="323935c9e1f07702be8b088c545084688c58b4c4830f737504790294c5b0fe7b"
Oct 30 04:32:07.369928 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:07.369906    4874 generic.go:356] "Generic (PLEG): container finished" podID="71b394ae-aff0-45fb-947e-05cd7bbd436a" containerID="ff8d8d0d40bba0e262adb0d9fb64f1b902805332711d8162aef8733d591d7cb9" exitCode=0
Oct 30 04:32:07.370029 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:07.369952    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr"
Oct 30 04:32:07.370029 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:07.369984    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr" event={"ID":"71b394ae-aff0-45fb-947e-05cd7bbd436a","Type":"ContainerDied","Data":"ff8d8d0d40bba0e262adb0d9fb64f1b902805332711d8162aef8733d591d7cb9"}
Oct 30 04:32:07.370145 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:07.370046    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr" event={"ID":"71b394ae-aff0-45fb-947e-05cd7bbd436a","Type":"ContainerDied","Data":"c6c18a8f16159797d29e57f747912276960060a44bc9a9bedc125c628b73bdae"}
Oct 30 04:32:07.378681 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:07.378665    4874 scope.go:122] "RemoveContainer" containerID="323935c9e1f07702be8b088c545084688c58b4c4830f737504790294c5b0fe7b"
Oct 30 04:32:07.378924 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:32:07.378905    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"323935c9e1f07702be8b088c545084688c58b4c4830f737504790294c5b0fe7b\": container with ID starting with 323935c9e1f07702be8b088c545084688c58b4c4830f737504790294c5b0fe7b not found: ID does not exist" containerID="323935c9e1f07702be8b088c545084688c58b4c4830f737504790294c5b0fe7b"
Oct 30 04:32:07.379001 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:07.378929    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"323935c9e1f07702be8b088c545084688c58b4c4830f737504790294c5b0fe7b"} err="failed to get container status \"323935c9e1f07702be8b088c545084688c58b4c4830f737504790294c5b0fe7b\": rpc error: code = NotFound desc = could not find container \"323935c9e1f07702be8b088c545084688c58b4c4830f737504790294c5b0fe7b\": container with ID starting with 323935c9e1f07702be8b088c545084688c58b4c4830f737504790294c5b0fe7b not found: ID does not exist"
Oct 30 04:32:07.379001 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:07.378945    4874 scope.go:122] "RemoveContainer" containerID="ff8d8d0d40bba0e262adb0d9fb64f1b902805332711d8162aef8733d591d7cb9"
Oct 30 04:32:07.388483 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:07.388466    4874 scope.go:122] "RemoveContainer" containerID="ff8d8d0d40bba0e262adb0d9fb64f1b902805332711d8162aef8733d591d7cb9"
Oct 30 04:32:07.388740 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:32:07.388721    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff8d8d0d40bba0e262adb0d9fb64f1b902805332711d8162aef8733d591d7cb9\": container with ID starting with ff8d8d0d40bba0e262adb0d9fb64f1b902805332711d8162aef8733d591d7cb9 not found: ID does not exist" containerID="ff8d8d0d40bba0e262adb0d9fb64f1b902805332711d8162aef8733d591d7cb9"
Oct 30 04:32:07.388790 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:07.388747    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff8d8d0d40bba0e262adb0d9fb64f1b902805332711d8162aef8733d591d7cb9"} err="failed to get container status \"ff8d8d0d40bba0e262adb0d9fb64f1b902805332711d8162aef8733d591d7cb9\": rpc error: code = NotFound desc = could not find container \"ff8d8d0d40bba0e262adb0d9fb64f1b902805332711d8162aef8733d591d7cb9\": container with ID starting with ff8d8d0d40bba0e262adb0d9fb64f1b902805332711d8162aef8733d591d7cb9 not found: ID does not exist"
Oct 30 04:32:07.419438 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:07.416980    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr"]
Oct 30 04:32:07.428969 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:07.428948    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-71b161-575fcb688c-wdcgr"]
Oct 30 04:32:07.460122 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:07.460085    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-71b161-668476b448-8489m"]
Oct 30 04:32:07.463553 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:07.463532    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-71b161-668476b448-8489m"]
Oct 30 04:32:08.377744 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:08.377711    4874 generic.go:356] "Generic (PLEG): container finished" podID="3f12d804-4aab-4672-af70-0f668cbb63a9" containerID="bc48517e00eb1827516ad79a3321d8299a415c407134f7128bd98fcc68a7c69f" exitCode=0
Oct 30 04:32:08.377938 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:08.377778    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx" event={"ID":"3f12d804-4aab-4672-af70-0f668cbb63a9","Type":"ContainerDied","Data":"bc48517e00eb1827516ad79a3321d8299a415c407134f7128bd98fcc68a7c69f"}
Oct 30 04:32:08.378274 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:08.378221    4874 scope.go:122] "RemoveContainer" containerID="bc48517e00eb1827516ad79a3321d8299a415c407134f7128bd98fcc68a7c69f"
Oct 30 04:32:08.380352 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:08.380320    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:32:08.380450 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:08.380404    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:32:08.380500 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:08.380448    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:32:08.733226 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:08.733192    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71b394ae-aff0-45fb-947e-05cd7bbd436a" path="/var/lib/kubelet/pods/71b394ae-aff0-45fb-947e-05cd7bbd436a/volumes"
Oct 30 04:32:08.733808 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:08.733787    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96a65bb0-e452-4068-ba0e-7145782c3b1f" path="/var/lib/kubelet/pods/96a65bb0-e452-4068-ba0e-7145782c3b1f/volumes"
Oct 30 04:32:09.395250 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:09.395195    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx" event={"ID":"3f12d804-4aab-4672-af70-0f668cbb63a9","Type":"ContainerStarted","Data":"80ca450ce9bb30d6b506ca828ef04dcaa352b05230f4a46572afee9b3eae1ce7"}
Oct 30 04:32:09.740852 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:09.740819    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:32:09.743411 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:09.743377    4874 patch_prober.go:29] interesting pod/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 30 04:32:09.743530 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:09.743437    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx" podUID="3f12d804-4aab-4672-af70-0f668cbb63a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 30 04:32:10.739228 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:10.739196    4874 patch_prober.go:29] interesting pod/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 30 04:32:10.739421 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:10.739268    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx" podUID="3f12d804-4aab-4672-af70-0f668cbb63a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 30 04:32:11.549865 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:11.549829    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:32:11.739107 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:11.739067    4874 patch_prober.go:29] interesting pod/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 30 04:32:11.739301 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:11.739114    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx" podUID="3f12d804-4aab-4672-af70-0f668cbb63a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 30 04:32:11.852331 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:11.847204    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:32:12.738999 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:12.738967    4874 patch_prober.go:29] interesting pod/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 30 04:32:12.739399 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:12.739031    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx" podUID="3f12d804-4aab-4672-af70-0f668cbb63a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 30 04:32:13.744451 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:13.744411    4874 patch_prober.go:29] interesting pod/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 30 04:32:13.744963 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:13.744483    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx" podUID="3f12d804-4aab-4672-af70-0f668cbb63a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 30 04:32:14.739299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:14.739263    4874 patch_prober.go:29] interesting pod/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 30 04:32:14.739504 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:14.739325    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx" podUID="3f12d804-4aab-4672-af70-0f668cbb63a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 30 04:32:15.744170 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:15.744140    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:32:15.745428 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:15.745402    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:32:25.610401 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:32:25.610365    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:32:37.073858 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:37.073830    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz_85f9442e-5e24-4a41-bf74-89ffc2cf240a/payload-processing/0.log"
Oct 30 04:32:37.267846 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:37.267818    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz_85f9442e-5e24-4a41-bf74-89ffc2cf240a/payload-processing/0.log"
Oct 30 04:32:37.486527 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:37.486496    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q"]
Oct 30 04:32:37.486813 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:37.486752    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q" podUID="8ee1f84a-09e0-483f-aea8-10ad797b3201" containerName="maas-api" containerID="cri-o://cfde6e72ff02db4e42f6abca02663229d54987f4b481acd477f80124d6af2b59" gracePeriod=30
Oct 30 04:32:37.714897 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:37.714866    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6"]
Oct 30 04:32:37.715153 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:37.715065    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6" podUID="6b9e2a70-701b-42c7-967c-70101a99e94e" containerName="payload-processing" containerID="cri-o://b431354ad001a66327061da7fd7ff9de70d10f7df9076cab8cfacd60a1603a98" gracePeriod=30
Oct 30 04:32:37.732906 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:37.732882    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj"]
Oct 30 04:32:37.733112 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:37.733089    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj" podUID="0622771d-893d-40bf-8397-f3b06a4e62f1" containerName="payload-pre-processing" containerID="cri-o://243a0e13eed7fc7cb8446c7cadc00acb1893b71c8050cfe8c7e917002a706b0b" gracePeriod=30
Oct 30 04:32:37.857487 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:37.857463    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q"
Oct 30 04:32:37.923375 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:37.922604    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee1f84a-09e0-483f-aea8-10ad797b3201-kube-api-access-4n4mb\" (UniqueName: \"kubernetes.io/projected/8ee1f84a-09e0-483f-aea8-10ad797b3201-kube-api-access-4n4mb\") pod \"8ee1f84a-09e0-483f-aea8-10ad797b3201\" (UID: \"8ee1f84a-09e0-483f-aea8-10ad797b3201\") "
Oct 30 04:32:37.923375 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:37.922667    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ee1f84a-09e0-483f-aea8-10ad797b3201-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ee1f84a-09e0-483f-aea8-10ad797b3201-metrics-certs\") pod \"8ee1f84a-09e0-483f-aea8-10ad797b3201\" (UID: \"8ee1f84a-09e0-483f-aea8-10ad797b3201\") "
Oct 30 04:32:37.923375 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:37.922738    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ee1f84a-09e0-483f-aea8-10ad797b3201-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee1f84a-09e0-483f-aea8-10ad797b3201-openshift-service-ca\") pod \"8ee1f84a-09e0-483f-aea8-10ad797b3201\" (UID: \"8ee1f84a-09e0-483f-aea8-10ad797b3201\") "
Oct 30 04:32:37.923375 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:37.922825    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ee1f84a-09e0-483f-aea8-10ad797b3201-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ee1f84a-09e0-483f-aea8-10ad797b3201-maas-api-tls\") pod \"8ee1f84a-09e0-483f-aea8-10ad797b3201\" (UID: \"8ee1f84a-09e0-483f-aea8-10ad797b3201\") "
Oct 30 04:32:37.924083 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:37.924059    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee1f84a-09e0-483f-aea8-10ad797b3201-openshift-service-ca" pod "8ee1f84a-09e0-483f-aea8-10ad797b3201" (UID: "8ee1f84a-09e0-483f-aea8-10ad797b3201"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:32:37.926409 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:37.926380    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ee1f84a-09e0-483f-aea8-10ad797b3201-maas-api-tls" pod "8ee1f84a-09e0-483f-aea8-10ad797b3201" (UID: "8ee1f84a-09e0-483f-aea8-10ad797b3201"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:32:37.929586 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:37.929560    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ee1f84a-09e0-483f-aea8-10ad797b3201-metrics-certs" pod "8ee1f84a-09e0-483f-aea8-10ad797b3201" (UID: "8ee1f84a-09e0-483f-aea8-10ad797b3201"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:32:37.931683 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:37.931650    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee1f84a-09e0-483f-aea8-10ad797b3201-kube-api-access-4n4mb" pod "8ee1f84a-09e0-483f-aea8-10ad797b3201" (UID: "8ee1f84a-09e0-483f-aea8-10ad797b3201"). InnerVolumeSpecName "kube-api-access-4n4mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:32:38.003984 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.003952    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6"
Oct 30 04:32:38.024534 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.024502    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b9e2a70-701b-42c7-967c-70101a99e94e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b9e2a70-701b-42c7-967c-70101a99e94e-plugins-config-volume\") pod \"6b9e2a70-701b-42c7-967c-70101a99e94e\" (UID: \"6b9e2a70-701b-42c7-967c-70101a99e94e\") "
Oct 30 04:32:38.024744 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.024660    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b9e2a70-701b-42c7-967c-70101a99e94e-kube-api-access-xxrpq\" (UniqueName: \"kubernetes.io/projected/6b9e2a70-701b-42c7-967c-70101a99e94e-kube-api-access-xxrpq\") pod \"6b9e2a70-701b-42c7-967c-70101a99e94e\" (UID: \"6b9e2a70-701b-42c7-967c-70101a99e94e\") "
Oct 30 04:32:38.024975 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.024946    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b9e2a70-701b-42c7-967c-70101a99e94e-plugins-config-volume" pod "6b9e2a70-701b-42c7-967c-70101a99e94e" (UID: "6b9e2a70-701b-42c7-967c-70101a99e94e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:32:38.025176 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.025154    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n4mb\" (UniqueName: \"kubernetes.io/projected/8ee1f84a-09e0-483f-aea8-10ad797b3201-kube-api-access-4n4mb\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:32:38.025324 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.025180    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ee1f84a-09e0-483f-aea8-10ad797b3201-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:32:38.025324 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.025196    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee1f84a-09e0-483f-aea8-10ad797b3201-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:32:38.025324 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.025209    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ee1f84a-09e0-483f-aea8-10ad797b3201-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:32:38.025324 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.025222    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b9e2a70-701b-42c7-967c-70101a99e94e-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:32:38.058017 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.057947    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b9e2a70-701b-42c7-967c-70101a99e94e-kube-api-access-xxrpq" pod "6b9e2a70-701b-42c7-967c-70101a99e94e" (UID: "6b9e2a70-701b-42c7-967c-70101a99e94e"). InnerVolumeSpecName "kube-api-access-xxrpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:32:38.130321 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.128974    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxrpq\" (UniqueName: \"kubernetes.io/projected/6b9e2a70-701b-42c7-967c-70101a99e94e-kube-api-access-xxrpq\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:32:38.169172 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.169148    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj"
Oct 30 04:32:38.330827 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.330742    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0622771d-893d-40bf-8397-f3b06a4e62f1-kube-api-access-njqdh\" (UniqueName: \"kubernetes.io/projected/0622771d-893d-40bf-8397-f3b06a4e62f1-kube-api-access-njqdh\") pod \"0622771d-893d-40bf-8397-f3b06a4e62f1\" (UID: \"0622771d-893d-40bf-8397-f3b06a4e62f1\") "
Oct 30 04:32:38.330995 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.330861    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0622771d-893d-40bf-8397-f3b06a4e62f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0622771d-893d-40bf-8397-f3b06a4e62f1-plugins-config-volume\") pod \"0622771d-893d-40bf-8397-f3b06a4e62f1\" (UID: \"0622771d-893d-40bf-8397-f3b06a4e62f1\") "
Oct 30 04:32:38.331313 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.331272    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0622771d-893d-40bf-8397-f3b06a4e62f1-plugins-config-volume" pod "0622771d-893d-40bf-8397-f3b06a4e62f1" (UID: "0622771d-893d-40bf-8397-f3b06a4e62f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:32:38.331445 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.331357    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0622771d-893d-40bf-8397-f3b06a4e62f1-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:32:38.333455 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.333426    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0622771d-893d-40bf-8397-f3b06a4e62f1-kube-api-access-njqdh" pod "0622771d-893d-40bf-8397-f3b06a4e62f1" (UID: "0622771d-893d-40bf-8397-f3b06a4e62f1"). InnerVolumeSpecName "kube-api-access-njqdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:32:38.432345 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.432318    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njqdh\" (UniqueName: \"kubernetes.io/projected/0622771d-893d-40bf-8397-f3b06a4e62f1-kube-api-access-njqdh\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:32:38.597618 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.597527    4874 generic.go:356] "Generic (PLEG): container finished" podID="8ee1f84a-09e0-483f-aea8-10ad797b3201" containerID="cfde6e72ff02db4e42f6abca02663229d54987f4b481acd477f80124d6af2b59" exitCode=0
Oct 30 04:32:38.597865 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.597579    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q"
Oct 30 04:32:38.598066 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.597596    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q" event={"ID":"8ee1f84a-09e0-483f-aea8-10ad797b3201","Type":"ContainerDied","Data":"cfde6e72ff02db4e42f6abca02663229d54987f4b481acd477f80124d6af2b59"}
Oct 30 04:32:38.598226 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.598209    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q" event={"ID":"8ee1f84a-09e0-483f-aea8-10ad797b3201","Type":"ContainerDied","Data":"b072da8530b33fbaed3e6b8078108029f4b6dd56c2a76f29accacf088b405d30"}
Oct 30 04:32:38.598584 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.598380    4874 scope.go:122] "RemoveContainer" containerID="cfde6e72ff02db4e42f6abca02663229d54987f4b481acd477f80124d6af2b59"
Oct 30 04:32:38.601184 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.600731    4874 generic.go:356] "Generic (PLEG): container finished" podID="6b9e2a70-701b-42c7-967c-70101a99e94e" containerID="b431354ad001a66327061da7fd7ff9de70d10f7df9076cab8cfacd60a1603a98" exitCode=0
Oct 30 04:32:38.601184 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.600808    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6"
Oct 30 04:32:38.601184 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.600901    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6" event={"ID":"6b9e2a70-701b-42c7-967c-70101a99e94e","Type":"ContainerDied","Data":"b431354ad001a66327061da7fd7ff9de70d10f7df9076cab8cfacd60a1603a98"}
Oct 30 04:32:38.601774 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.601758    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6" event={"ID":"6b9e2a70-701b-42c7-967c-70101a99e94e","Type":"ContainerDied","Data":"506ce4190762eb9840d71edb73d7805dc2f0d02d74febf0a62a6020dc40ac063"}
Oct 30 04:32:38.602786 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.602574    4874 generic.go:356] "Generic (PLEG): container finished" podID="0622771d-893d-40bf-8397-f3b06a4e62f1" containerID="243a0e13eed7fc7cb8446c7cadc00acb1893b71c8050cfe8c7e917002a706b0b" exitCode=0
Oct 30 04:32:38.602786 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.602712    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj" event={"ID":"0622771d-893d-40bf-8397-f3b06a4e62f1","Type":"ContainerDied","Data":"243a0e13eed7fc7cb8446c7cadc00acb1893b71c8050cfe8c7e917002a706b0b"}
Oct 30 04:32:38.602786 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.602730    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj"
Oct 30 04:32:38.602786 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.602747    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj" event={"ID":"0622771d-893d-40bf-8397-f3b06a4e62f1","Type":"ContainerDied","Data":"3ffefc8721823f99ede7962f027cc4862c01eab721e09a872381680fcf4eda24"}
Oct 30 04:32:38.614183 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.614160    4874 scope.go:122] "RemoveContainer" containerID="cfde6e72ff02db4e42f6abca02663229d54987f4b481acd477f80124d6af2b59"
Oct 30 04:32:38.614513 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:32:38.614490    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfde6e72ff02db4e42f6abca02663229d54987f4b481acd477f80124d6af2b59\": container with ID starting with cfde6e72ff02db4e42f6abca02663229d54987f4b481acd477f80124d6af2b59 not found: ID does not exist" containerID="cfde6e72ff02db4e42f6abca02663229d54987f4b481acd477f80124d6af2b59"
Oct 30 04:32:38.614619 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.614524    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfde6e72ff02db4e42f6abca02663229d54987f4b481acd477f80124d6af2b59"} err="failed to get container status \"cfde6e72ff02db4e42f6abca02663229d54987f4b481acd477f80124d6af2b59\": rpc error: code = NotFound desc = could not find container \"cfde6e72ff02db4e42f6abca02663229d54987f4b481acd477f80124d6af2b59\": container with ID starting with cfde6e72ff02db4e42f6abca02663229d54987f4b481acd477f80124d6af2b59 not found: ID does not exist"
Oct 30 04:32:38.614619 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.614549    4874 scope.go:122] "RemoveContainer" containerID="b431354ad001a66327061da7fd7ff9de70d10f7df9076cab8cfacd60a1603a98"
Oct 30 04:32:38.623852 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.623830    4874 scope.go:122] "RemoveContainer" containerID="b431354ad001a66327061da7fd7ff9de70d10f7df9076cab8cfacd60a1603a98"
Oct 30 04:32:38.624111 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:32:38.624092    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b431354ad001a66327061da7fd7ff9de70d10f7df9076cab8cfacd60a1603a98\": container with ID starting with b431354ad001a66327061da7fd7ff9de70d10f7df9076cab8cfacd60a1603a98 not found: ID does not exist" containerID="b431354ad001a66327061da7fd7ff9de70d10f7df9076cab8cfacd60a1603a98"
Oct 30 04:32:38.624164 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.624118    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b431354ad001a66327061da7fd7ff9de70d10f7df9076cab8cfacd60a1603a98"} err="failed to get container status \"b431354ad001a66327061da7fd7ff9de70d10f7df9076cab8cfacd60a1603a98\": rpc error: code = NotFound desc = could not find container \"b431354ad001a66327061da7fd7ff9de70d10f7df9076cab8cfacd60a1603a98\": container with ID starting with b431354ad001a66327061da7fd7ff9de70d10f7df9076cab8cfacd60a1603a98 not found: ID does not exist"
Oct 30 04:32:38.624164 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.624137    4874 scope.go:122] "RemoveContainer" containerID="243a0e13eed7fc7cb8446c7cadc00acb1893b71c8050cfe8c7e917002a706b0b"
Oct 30 04:32:38.632716 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.632699    4874 scope.go:122] "RemoveContainer" containerID="243a0e13eed7fc7cb8446c7cadc00acb1893b71c8050cfe8c7e917002a706b0b"
Oct 30 04:32:38.632979 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:32:38.632958    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"243a0e13eed7fc7cb8446c7cadc00acb1893b71c8050cfe8c7e917002a706b0b\": container with ID starting with 243a0e13eed7fc7cb8446c7cadc00acb1893b71c8050cfe8c7e917002a706b0b not found: ID does not exist" containerID="243a0e13eed7fc7cb8446c7cadc00acb1893b71c8050cfe8c7e917002a706b0b"
Oct 30 04:32:38.633017 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.632986    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"243a0e13eed7fc7cb8446c7cadc00acb1893b71c8050cfe8c7e917002a706b0b"} err="failed to get container status \"243a0e13eed7fc7cb8446c7cadc00acb1893b71c8050cfe8c7e917002a706b0b\": rpc error: code = NotFound desc = could not find container \"243a0e13eed7fc7cb8446c7cadc00acb1893b71c8050cfe8c7e917002a706b0b\": container with ID starting with 243a0e13eed7fc7cb8446c7cadc00acb1893b71c8050cfe8c7e917002a706b0b not found: ID does not exist"
Oct 30 04:32:38.653785 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.653755    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q"]
Oct 30 04:32:38.663000 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.662971    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f5508-857769d577-wnc2q"]
Oct 30 04:32:38.707883 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.707854    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6"]
Oct 30 04:32:38.714315 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.714293    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3f5508-69d86d8b84-nvfm6"]
Oct 30 04:32:38.729666 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.729643    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b9e2a70-701b-42c7-967c-70101a99e94e" path="/var/lib/kubelet/pods/6b9e2a70-701b-42c7-967c-70101a99e94e/volumes"
Oct 30 04:32:38.730076 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.730063    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ee1f84a-09e0-483f-aea8-10ad797b3201" path="/var/lib/kubelet/pods/8ee1f84a-09e0-483f-aea8-10ad797b3201/volumes"
Oct 30 04:32:38.755514 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.755491    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj"]
Oct 30 04:32:38.759313 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:38.759289    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3f5508-676f7884ff-wdjwj"]
Oct 30 04:32:40.730686 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:40.730650    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0622771d-893d-40bf-8397-f3b06a4e62f1" path="/var/lib/kubelet/pods/0622771d-893d-40bf-8397-f3b06a4e62f1/volumes"
Oct 30 04:32:43.248658 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:43.248622    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"]
Oct 30 04:32:48.127462 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:48.127429    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:32:55.609259 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:32:55.609211    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:32:58.269194 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.269127    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4" podUID="095312cb-be01-4f88-8cc0-87eecca32f5d" containerName="main" containerID="cri-o://bf06354d826110001d5dd86841f10bbb45449eb1c44e15eaa85ce9f209a6beb2" gracePeriod=45
Oct 30 04:32:58.515641 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.515615    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:32:58.633994 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.633911    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-home\" (UniqueName: \"kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-home\") pod \"095312cb-be01-4f88-8cc0-87eecca32f5d\" (UID: \"095312cb-be01-4f88-8cc0-87eecca32f5d\") "
Oct 30 04:32:58.634159 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.634016    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/095312cb-be01-4f88-8cc0-87eecca32f5d-tls-certs\" (UniqueName: \"kubernetes.io/secret/095312cb-be01-4f88-8cc0-87eecca32f5d-tls-certs\") pod \"095312cb-be01-4f88-8cc0-87eecca32f5d\" (UID: \"095312cb-be01-4f88-8cc0-87eecca32f5d\") "
Oct 30 04:32:58.634159 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.634064    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-tmp-dir\") pod \"095312cb-be01-4f88-8cc0-87eecca32f5d\" (UID: \"095312cb-be01-4f88-8cc0-87eecca32f5d\") "
Oct 30 04:32:58.634159 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.634137    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-dshm\") pod \"095312cb-be01-4f88-8cc0-87eecca32f5d\" (UID: \"095312cb-be01-4f88-8cc0-87eecca32f5d\") "
Oct 30 04:32:58.634356 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.634163    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-home" pod "095312cb-be01-4f88-8cc0-87eecca32f5d" (UID: "095312cb-be01-4f88-8cc0-87eecca32f5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:32:58.634356 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.634171    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/095312cb-be01-4f88-8cc0-87eecca32f5d-kube-api-access-gq8q6\" (UniqueName: \"kubernetes.io/projected/095312cb-be01-4f88-8cc0-87eecca32f5d-kube-api-access-gq8q6\") pod \"095312cb-be01-4f88-8cc0-87eecca32f5d\" (UID: \"095312cb-be01-4f88-8cc0-87eecca32f5d\") "
Oct 30 04:32:58.634356 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.634295    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-model-cache\") pod \"095312cb-be01-4f88-8cc0-87eecca32f5d\" (UID: \"095312cb-be01-4f88-8cc0-87eecca32f5d\") "
Oct 30 04:32:58.634496 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.634382    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-tmp-dir" pod "095312cb-be01-4f88-8cc0-87eecca32f5d" (UID: "095312cb-be01-4f88-8cc0-87eecca32f5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:32:58.634570 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.634543    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-model-cache" pod "095312cb-be01-4f88-8cc0-87eecca32f5d" (UID: "095312cb-be01-4f88-8cc0-87eecca32f5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:32:58.634849 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.634829    4874 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-tmp-dir\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:32:58.634849 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.634849    4874 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-model-cache\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:32:58.634993 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.634861    4874 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-home\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:32:58.636358 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.636334    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095312cb-be01-4f88-8cc0-87eecca32f5d-kube-api-access-gq8q6" pod "095312cb-be01-4f88-8cc0-87eecca32f5d" (UID: "095312cb-be01-4f88-8cc0-87eecca32f5d"). InnerVolumeSpecName "kube-api-access-gq8q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:32:58.636629 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.636604    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/095312cb-be01-4f88-8cc0-87eecca32f5d-tls-certs" pod "095312cb-be01-4f88-8cc0-87eecca32f5d" (UID: "095312cb-be01-4f88-8cc0-87eecca32f5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:32:58.636724 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.636602    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-dshm" pod "095312cb-be01-4f88-8cc0-87eecca32f5d" (UID: "095312cb-be01-4f88-8cc0-87eecca32f5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:32:58.735709 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.735679    4874 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095312cb-be01-4f88-8cc0-87eecca32f5d-tls-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:32:58.735709 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.735707    4874 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095312cb-be01-4f88-8cc0-87eecca32f5d-dshm\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:32:58.735709 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.735717    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq8q6\" (UniqueName: \"kubernetes.io/projected/095312cb-be01-4f88-8cc0-87eecca32f5d-kube-api-access-gq8q6\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:32:58.736822 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.736794    4874 generic.go:356] "Generic (PLEG): container finished" podID="095312cb-be01-4f88-8cc0-87eecca32f5d" containerID="bf06354d826110001d5dd86841f10bbb45449eb1c44e15eaa85ce9f209a6beb2" exitCode=0
Oct 30 04:32:58.736945 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.736829    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"
Oct 30 04:32:58.736945 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.736891    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4" event={"ID":"095312cb-be01-4f88-8cc0-87eecca32f5d","Type":"ContainerDied","Data":"bf06354d826110001d5dd86841f10bbb45449eb1c44e15eaa85ce9f209a6beb2"}
Oct 30 04:32:58.736945 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.736933    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4" event={"ID":"095312cb-be01-4f88-8cc0-87eecca32f5d","Type":"ContainerDied","Data":"83daf016d9aff4ff35daad712412cbd364952914a582360786070cac93e6f024"}
Oct 30 04:32:58.737045 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.736955    4874 scope.go:122] "RemoveContainer" containerID="bf06354d826110001d5dd86841f10bbb45449eb1c44e15eaa85ce9f209a6beb2"
Oct 30 04:32:58.749438 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.749414    4874 scope.go:122] "RemoveContainer" containerID="bf06354d826110001d5dd86841f10bbb45449eb1c44e15eaa85ce9f209a6beb2"
Oct 30 04:32:58.749749 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:32:58.749723    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf06354d826110001d5dd86841f10bbb45449eb1c44e15eaa85ce9f209a6beb2\": container with ID starting with bf06354d826110001d5dd86841f10bbb45449eb1c44e15eaa85ce9f209a6beb2 not found: ID does not exist" containerID="bf06354d826110001d5dd86841f10bbb45449eb1c44e15eaa85ce9f209a6beb2"
Oct 30 04:32:58.749826 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.749759    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf06354d826110001d5dd86841f10bbb45449eb1c44e15eaa85ce9f209a6beb2"} err="failed to get container status \"bf06354d826110001d5dd86841f10bbb45449eb1c44e15eaa85ce9f209a6beb2\": rpc error: code = NotFound desc = could not find container \"bf06354d826110001d5dd86841f10bbb45449eb1c44e15eaa85ce9f209a6beb2\": container with ID starting with bf06354d826110001d5dd86841f10bbb45449eb1c44e15eaa85ce9f209a6beb2 not found: ID does not exist"
Oct 30 04:32:58.786321 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.786294    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"]
Oct 30 04:32:58.789818 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:58.789800    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-f7a36d/ipp-route-f7a36d-kserve-65bd4f59fc-25xn4"]
Oct 30 04:32:59.203101 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:59.203054    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn"]
Oct 30 04:32:59.203385 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:59.203358    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn" podUID="3d8911a9-d86c-4efb-99dc-156a74ad8628" containerName="maas-api" containerID="cri-o://d71e182b92a06babbb089ba9d3c2550cae60410eafcaa8029a37ae556411295c" gracePeriod=30
Oct 30 04:32:59.488953 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:59.488924    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn"
Oct 30 04:32:59.647861 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:59.647834    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d8911a9-d86c-4efb-99dc-156a74ad8628-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d8911a9-d86c-4efb-99dc-156a74ad8628-maas-api-tls\") pod \"3d8911a9-d86c-4efb-99dc-156a74ad8628\" (UID: \"3d8911a9-d86c-4efb-99dc-156a74ad8628\") "
Oct 30 04:32:59.648029 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:59.647886    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d8911a9-d86c-4efb-99dc-156a74ad8628-kube-api-access-p2ln7\" (UniqueName: \"kubernetes.io/projected/3d8911a9-d86c-4efb-99dc-156a74ad8628-kube-api-access-p2ln7\") pod \"3d8911a9-d86c-4efb-99dc-156a74ad8628\" (UID: \"3d8911a9-d86c-4efb-99dc-156a74ad8628\") "
Oct 30 04:32:59.648029 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:59.647904    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d8911a9-d86c-4efb-99dc-156a74ad8628-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8911a9-d86c-4efb-99dc-156a74ad8628-metrics-certs\") pod \"3d8911a9-d86c-4efb-99dc-156a74ad8628\" (UID: \"3d8911a9-d86c-4efb-99dc-156a74ad8628\") "
Oct 30 04:32:59.648029 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:59.648006    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d8911a9-d86c-4efb-99dc-156a74ad8628-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d8911a9-d86c-4efb-99dc-156a74ad8628-openshift-service-ca\") pod \"3d8911a9-d86c-4efb-99dc-156a74ad8628\" (UID: \"3d8911a9-d86c-4efb-99dc-156a74ad8628\") "
Oct 30 04:32:59.648427 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:59.648394    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8911a9-d86c-4efb-99dc-156a74ad8628-openshift-service-ca" pod "3d8911a9-d86c-4efb-99dc-156a74ad8628" (UID: "3d8911a9-d86c-4efb-99dc-156a74ad8628"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:32:59.650029 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:59.650003    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d8911a9-d86c-4efb-99dc-156a74ad8628-maas-api-tls" pod "3d8911a9-d86c-4efb-99dc-156a74ad8628" (UID: "3d8911a9-d86c-4efb-99dc-156a74ad8628"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:32:59.650126 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:59.650058    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d8911a9-d86c-4efb-99dc-156a74ad8628-metrics-certs" pod "3d8911a9-d86c-4efb-99dc-156a74ad8628" (UID: "3d8911a9-d86c-4efb-99dc-156a74ad8628"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:32:59.650126 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:59.650086    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8911a9-d86c-4efb-99dc-156a74ad8628-kube-api-access-p2ln7" pod "3d8911a9-d86c-4efb-99dc-156a74ad8628" (UID: "3d8911a9-d86c-4efb-99dc-156a74ad8628"). InnerVolumeSpecName "kube-api-access-p2ln7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:32:59.772262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:59.755827    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d8911a9-d86c-4efb-99dc-156a74ad8628-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:32:59.772262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:59.755863    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d8911a9-d86c-4efb-99dc-156a74ad8628-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:32:59.772262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:59.755876    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2ln7\" (UniqueName: \"kubernetes.io/projected/3d8911a9-d86c-4efb-99dc-156a74ad8628-kube-api-access-p2ln7\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:32:59.772262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:59.755889    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8911a9-d86c-4efb-99dc-156a74ad8628-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:32:59.870288 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:59.867342    4874 generic.go:356] "Generic (PLEG): container finished" podID="3d8911a9-d86c-4efb-99dc-156a74ad8628" containerID="d71e182b92a06babbb089ba9d3c2550cae60410eafcaa8029a37ae556411295c" exitCode=0
Oct 30 04:32:59.870288 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:59.867381    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn"
Oct 30 04:32:59.870288 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:59.867405    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn" event={"ID":"3d8911a9-d86c-4efb-99dc-156a74ad8628","Type":"ContainerDied","Data":"d71e182b92a06babbb089ba9d3c2550cae60410eafcaa8029a37ae556411295c"}
Oct 30 04:32:59.870288 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:59.867458    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn" event={"ID":"3d8911a9-d86c-4efb-99dc-156a74ad8628","Type":"ContainerDied","Data":"d0c20c1c310a503a0e10df7f56566c8284470389693fb23438108a3248533d16"}
Oct 30 04:32:59.870288 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:59.867487    4874 scope.go:122] "RemoveContainer" containerID="d71e182b92a06babbb089ba9d3c2550cae60410eafcaa8029a37ae556411295c"
Oct 30 04:32:59.885374 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:59.885347    4874 scope.go:122] "RemoveContainer" containerID="d71e182b92a06babbb089ba9d3c2550cae60410eafcaa8029a37ae556411295c"
Oct 30 04:32:59.885761 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:32:59.885700    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d71e182b92a06babbb089ba9d3c2550cae60410eafcaa8029a37ae556411295c\": container with ID starting with d71e182b92a06babbb089ba9d3c2550cae60410eafcaa8029a37ae556411295c not found: ID does not exist" containerID="d71e182b92a06babbb089ba9d3c2550cae60410eafcaa8029a37ae556411295c"
Oct 30 04:32:59.885761 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:59.885729    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d71e182b92a06babbb089ba9d3c2550cae60410eafcaa8029a37ae556411295c"} err="failed to get container status \"d71e182b92a06babbb089ba9d3c2550cae60410eafcaa8029a37ae556411295c\": rpc error: code = NotFound desc = could not find container \"d71e182b92a06babbb089ba9d3c2550cae60410eafcaa8029a37ae556411295c\": container with ID starting with d71e182b92a06babbb089ba9d3c2550cae60410eafcaa8029a37ae556411295c not found: ID does not exist"
Oct 30 04:32:59.918179 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:59.918144    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn"]
Oct 30 04:32:59.924814 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:32:59.924787    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a777c-57f685bc9b-4rrxn"]
Oct 30 04:33:00.733334 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:00.733302    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="095312cb-be01-4f88-8cc0-87eecca32f5d" path="/var/lib/kubelet/pods/095312cb-be01-4f88-8cc0-87eecca32f5d/volumes"
Oct 30 04:33:00.733759 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:00.733745    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d8911a9-d86c-4efb-99dc-156a74ad8628" path="/var/lib/kubelet/pods/3d8911a9-d86c-4efb-99dc-156a74ad8628/volumes"
Oct 30 04:33:00.925729 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:00.925702    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 04:33:00.957520 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:00.957499    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 04:33:06.188923 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:06.188879    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"]
Oct 30 04:33:06.189432 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:06.189120    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v" podUID="1f0cc4fd-bf2b-4642-ab23-e6649237f0fb" containerName="istio-proxy" containerID="cri-o://ebf13b275d089f49ed0872ebffe0c7c84e05f58637835d6643cb1b52a35234d5" gracePeriod=30
Oct 30 04:33:10.063120 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:10.063083    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:33:11.464267 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.464195    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:33:11.521196 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.521163    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-envoy\") pod \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") "
Oct 30 04:33:11.521398 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.521220    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-credential-socket\") pod \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") "
Oct 30 04:33:11.521398 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.521298    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-data\") pod \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") "
Oct 30 04:33:11.521398 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.521367    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-workload-socket\") pod \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") "
Oct 30 04:33:11.521550 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.521415    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istiod-ca-cert\") pod \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") "
Oct 30 04:33:11.521550 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.521456    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-token\" (UniqueName: \"kubernetes.io/projected/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-token\") pod \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") "
Oct 30 04:33:11.521638 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.521561    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-podinfo\") pod \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") "
Oct 30 04:33:11.521638 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.521596    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-workload-certs\") pod \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") "
Oct 30 04:33:11.521729 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.521705    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-kube-api-access-58hcq\" (UniqueName: \"kubernetes.io/projected/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-kube-api-access-58hcq\") pod \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\" (UID: \"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb\") "
Oct 30 04:33:11.522175 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.522063    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-workload-socket" pod "1f0cc4fd-bf2b-4642-ab23-e6649237f0fb" (UID: "1f0cc4fd-bf2b-4642-ab23-e6649237f0fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:33:11.522406 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.522379    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:11.522944 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.522891    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-data" pod "1f0cc4fd-bf2b-4642-ab23-e6649237f0fb" (UID: "1f0cc4fd-bf2b-4642-ab23-e6649237f0fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:33:11.522944 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.522922    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-credential-socket" pod "1f0cc4fd-bf2b-4642-ab23-e6649237f0fb" (UID: "1f0cc4fd-bf2b-4642-ab23-e6649237f0fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:33:11.523141 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.522808    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istiod-ca-cert" pod "1f0cc4fd-bf2b-4642-ab23-e6649237f0fb" (UID: "1f0cc4fd-bf2b-4642-ab23-e6649237f0fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:33:11.523141 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.523011    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-workload-certs" pod "1f0cc4fd-bf2b-4642-ab23-e6649237f0fb" (UID: "1f0cc4fd-bf2b-4642-ab23-e6649237f0fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:33:11.524781 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.524753    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-kube-api-access-58hcq" pod "1f0cc4fd-bf2b-4642-ab23-e6649237f0fb" (UID: "1f0cc4fd-bf2b-4642-ab23-e6649237f0fb"). InnerVolumeSpecName "kube-api-access-58hcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:33:11.525430 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.525394    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-envoy" pod "1f0cc4fd-bf2b-4642-ab23-e6649237f0fb" (UID: "1f0cc4fd-bf2b-4642-ab23-e6649237f0fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:33:11.526408 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.526384    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-token" pod "1f0cc4fd-bf2b-4642-ab23-e6649237f0fb" (UID: "1f0cc4fd-bf2b-4642-ab23-e6649237f0fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:33:11.528055 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.527812    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-podinfo" pod "1f0cc4fd-bf2b-4642-ab23-e6649237f0fb" (UID: "1f0cc4fd-bf2b-4642-ab23-e6649237f0fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:33:11.623632 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.623598    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:11.623632 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.623626    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:11.623632 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.623635    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:11.623931 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.623643    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:11.623931 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.623652    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58hcq\" (UniqueName: \"kubernetes.io/projected/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-kube-api-access-58hcq\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:11.623931 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.623661    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:11.623931 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.623669    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:11.623931 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.623677    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:11.938840 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.938802    4874 generic.go:356] "Generic (PLEG): container finished" podID="1f0cc4fd-bf2b-4642-ab23-e6649237f0fb" containerID="ebf13b275d089f49ed0872ebffe0c7c84e05f58637835d6643cb1b52a35234d5" exitCode=0
Oct 30 04:33:11.939019 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.938872    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"
Oct 30 04:33:11.939019 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.938886    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v" event={"ID":"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb","Type":"ContainerDied","Data":"ebf13b275d089f49ed0872ebffe0c7c84e05f58637835d6643cb1b52a35234d5"}
Oct 30 04:33:11.939019 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.938939    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v" event={"ID":"1f0cc4fd-bf2b-4642-ab23-e6649237f0fb","Type":"ContainerDied","Data":"d230364bce23af75a4ffcf47c41683e7aab5456a8ef361aa52566ee9dd17c235"}
Oct 30 04:33:11.939019 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.938960    4874 scope.go:122] "RemoveContainer" containerID="ebf13b275d089f49ed0872ebffe0c7c84e05f58637835d6643cb1b52a35234d5"
Oct 30 04:33:11.948700 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.948681    4874 scope.go:122] "RemoveContainer" containerID="f640d6169b90ad661207d8685aba0f6f09fdfe54c9f5e0896727909458157434"
Oct 30 04:33:11.959471 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.959408    4874 scope.go:122] "RemoveContainer" containerID="ebf13b275d089f49ed0872ebffe0c7c84e05f58637835d6643cb1b52a35234d5"
Oct 30 04:33:11.959730 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:33:11.959706    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebf13b275d089f49ed0872ebffe0c7c84e05f58637835d6643cb1b52a35234d5\": container with ID starting with ebf13b275d089f49ed0872ebffe0c7c84e05f58637835d6643cb1b52a35234d5 not found: ID does not exist" containerID="ebf13b275d089f49ed0872ebffe0c7c84e05f58637835d6643cb1b52a35234d5"
Oct 30 04:33:11.959781 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.959743    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf13b275d089f49ed0872ebffe0c7c84e05f58637835d6643cb1b52a35234d5"} err="failed to get container status \"ebf13b275d089f49ed0872ebffe0c7c84e05f58637835d6643cb1b52a35234d5\": rpc error: code = NotFound desc = could not find container \"ebf13b275d089f49ed0872ebffe0c7c84e05f58637835d6643cb1b52a35234d5\": container with ID starting with ebf13b275d089f49ed0872ebffe0c7c84e05f58637835d6643cb1b52a35234d5 not found: ID does not exist"
Oct 30 04:33:11.959781 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.959766    4874 scope.go:122] "RemoveContainer" containerID="f640d6169b90ad661207d8685aba0f6f09fdfe54c9f5e0896727909458157434"
Oct 30 04:33:11.960036 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:33:11.960012    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f640d6169b90ad661207d8685aba0f6f09fdfe54c9f5e0896727909458157434\": container with ID starting with f640d6169b90ad661207d8685aba0f6f09fdfe54c9f5e0896727909458157434 not found: ID does not exist" containerID="f640d6169b90ad661207d8685aba0f6f09fdfe54c9f5e0896727909458157434"
Oct 30 04:33:11.960079 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.960042    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f640d6169b90ad661207d8685aba0f6f09fdfe54c9f5e0896727909458157434"} err="failed to get container status \"f640d6169b90ad661207d8685aba0f6f09fdfe54c9f5e0896727909458157434\": rpc error: code = NotFound desc = could not find container \"f640d6169b90ad661207d8685aba0f6f09fdfe54c9f5e0896727909458157434\": container with ID starting with f640d6169b90ad661207d8685aba0f6f09fdfe54c9f5e0896727909458157434 not found: ID does not exist"
Oct 30 04:33:11.989003 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.988972    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"]
Oct 30 04:33:11.996033 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:11.996003    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6a777c-openshift-default-597bb4b5dc-v6p4v"]
Oct 30 04:33:12.729003 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:12.728972    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f0cc4fd-bf2b-4642-ab23-e6649237f0fb" path="/var/lib/kubelet/pods/1f0cc4fd-bf2b-4642-ab23-e6649237f0fb/volumes"
Oct 30 04:33:21.774963 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:21.774874    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"]
Oct 30 04:33:21.775452 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:21.775099    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2" podUID="908b541f-e807-4403-98c4-41e5cea32837" containerName="istio-proxy" containerID="cri-o://3869f7e77e1916369207bd75eb0ef2b7e0db8adb777474de44bfa3fe30828802" gracePeriod=30
Oct 30 04:33:21.786484 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:21.786458    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"]
Oct 30 04:33:21.786669 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:21.786646    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg" podUID="157dc9a9-a3f1-4de0-9787-22207838b7b7" containerName="istio-proxy" containerID="cri-o://7d7d7e235b43707b4c5ab8e8aca9330cc6443e91cec5ce978e6599eb16b2696e" gracePeriod=30
Oct 30 04:33:21.789682 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:21.789655    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"]
Oct 30 04:33:21.789894 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:21.789869    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m" podUID="0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1" containerName="istio-proxy" containerID="cri-o://8f6812d18fb2347359ebaca20b438759a88954b5e44a28b32681a95b2df8301c" gracePeriod=30
Oct 30 04:33:23.413821 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.413783    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"]
Oct 30 04:33:23.414571 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414549    4874 cpu_manager.go:414] "containerMap: removing container" podUID="1f0cc4fd-bf2b-4642-ab23-e6649237f0fb" containerName="istio-proxy"
Oct 30 04:33:23.414633 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414576    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f0cc4fd-bf2b-4642-ab23-e6649237f0fb" containerName="istio-proxy"
Oct 30 04:33:23.414633 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414603    4874 cpu_manager.go:414] "containerMap: removing container" podUID="71b394ae-aff0-45fb-947e-05cd7bbd436a" containerName="payload-pre-processing"
Oct 30 04:33:23.414633 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414614    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b394ae-aff0-45fb-947e-05cd7bbd436a" containerName="payload-pre-processing"
Oct 30 04:33:23.414633 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414628    4874 cpu_manager.go:414] "containerMap: removing container" podUID="96a65bb0-e452-4068-ba0e-7145782c3b1f" containerName="payload-processing"
Oct 30 04:33:23.414762 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414636    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a65bb0-e452-4068-ba0e-7145782c3b1f" containerName="payload-processing"
Oct 30 04:33:23.414762 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414655    4874 cpu_manager.go:414] "containerMap: removing container" podUID="0622771d-893d-40bf-8397-f3b06a4e62f1" containerName="payload-pre-processing"
Oct 30 04:33:23.414762 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414663    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0622771d-893d-40bf-8397-f3b06a4e62f1" containerName="payload-pre-processing"
Oct 30 04:33:23.414762 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414682    4874 cpu_manager.go:414] "containerMap: removing container" podUID="6b9e2a70-701b-42c7-967c-70101a99e94e" containerName="payload-processing"
Oct 30 04:33:23.414762 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414689    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b9e2a70-701b-42c7-967c-70101a99e94e" containerName="payload-processing"
Oct 30 04:33:23.414762 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414707    4874 cpu_manager.go:414] "containerMap: removing container" podUID="1d4d2573-179f-46bc-be32-e3dcc140437f" containerName="payload-pre-processing"
Oct 30 04:33:23.414762 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414714    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d4d2573-179f-46bc-be32-e3dcc140437f" containerName="payload-pre-processing"
Oct 30 04:33:23.414762 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414728    4874 cpu_manager.go:414] "containerMap: removing container" podUID="3d8911a9-d86c-4efb-99dc-156a74ad8628" containerName="maas-api"
Oct 30 04:33:23.414762 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414735    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d8911a9-d86c-4efb-99dc-156a74ad8628" containerName="maas-api"
Oct 30 04:33:23.415034 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414768    4874 cpu_manager.go:414] "containerMap: removing container" podUID="095312cb-be01-4f88-8cc0-87eecca32f5d" containerName="main"
Oct 30 04:33:23.415034 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414775    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="095312cb-be01-4f88-8cc0-87eecca32f5d" containerName="main"
Oct 30 04:33:23.415034 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414789    4874 cpu_manager.go:414] "containerMap: removing container" podUID="8ee1f84a-09e0-483f-aea8-10ad797b3201" containerName="maas-api"
Oct 30 04:33:23.415034 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414795    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee1f84a-09e0-483f-aea8-10ad797b3201" containerName="maas-api"
Oct 30 04:33:23.415034 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414904    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b9e2a70-701b-42c7-967c-70101a99e94e" containerName="payload-processing"
Oct 30 04:33:23.415034 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414919    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d4d2573-179f-46bc-be32-e3dcc140437f" containerName="payload-pre-processing"
Oct 30 04:33:23.415034 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414930    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f0cc4fd-bf2b-4642-ab23-e6649237f0fb" containerName="istio-proxy"
Oct 30 04:33:23.415034 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414939    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d8911a9-d86c-4efb-99dc-156a74ad8628" containerName="maas-api"
Oct 30 04:33:23.415034 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414950    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="095312cb-be01-4f88-8cc0-87eecca32f5d" containerName="main"
Oct 30 04:33:23.415034 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414959    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee1f84a-09e0-483f-aea8-10ad797b3201" containerName="maas-api"
Oct 30 04:33:23.415034 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414971    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f0cc4fd-bf2b-4642-ab23-e6649237f0fb" containerName="istio-proxy"
Oct 30 04:33:23.415034 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414983    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b394ae-aff0-45fb-947e-05cd7bbd436a" containerName="payload-pre-processing"
Oct 30 04:33:23.415034 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.414991    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a65bb0-e452-4068-ba0e-7145782c3b1f" containerName="payload-processing"
Oct 30 04:33:23.415034 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.415002    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="0622771d-893d-40bf-8397-f3b06a4e62f1" containerName="payload-pre-processing"
Oct 30 04:33:23.417665 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.417646    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.446328 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.446301    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b4708a-openshift-default-dockercfg-t5rn6\""
Oct 30 04:33:23.449087 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.449058    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"]
Oct 30 04:33:23.462836 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.462805    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.462962 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.462908    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.462962 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.462957    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.463134 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.462974    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.463134 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.463019    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.463134 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.463134    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.463398 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.463166    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z9w6\" (UniqueName: \"kubernetes.io/projected/4ccf3fdf-09b2-48b2-a270-2500fe34b892-kube-api-access-8z9w6\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.463398 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.463196    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.463398 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.463225    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.564111 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.564076    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.564307 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.564121    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.564307 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.564138    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.564307 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.564162    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.564307 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.564214    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.564307 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.564278    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z9w6\" (UniqueName: \"kubernetes.io/projected/4ccf3fdf-09b2-48b2-a270-2500fe34b892-kube-api-access-8z9w6\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.564583 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.564311    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.564583 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.564349    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.564583 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.564416    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.564583 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.564526    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.564830 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.564804    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.564977 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.564945    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.565096 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.565073    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.565290 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.565266    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.566969 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.566947    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.567311 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.567293    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.586709 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.586682    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z9w6\" (UniqueName: \"kubernetes.io/projected/4ccf3fdf-09b2-48b2-a270-2500fe34b892-kube-api-access-8z9w6\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.587436 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.587415    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:23.738025 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:23.732998    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:24.057035 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:24.057012    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"]
Oct 30 04:33:24.058395 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:33:24.058373    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ccf3fdf_09b2_48b2_a270_2500fe34b892.slice/crio-8fccff1f6d20d080bb8c69f93343282cebf55836f874d493f6d32a3a2cf88ee8 WatchSource:0}: Error finding container 8fccff1f6d20d080bb8c69f93343282cebf55836f874d493f6d32a3a2cf88ee8: Status 404 returned error can't find the container with id 8fccff1f6d20d080bb8c69f93343282cebf55836f874d493f6d32a3a2cf88ee8
Oct 30 04:33:24.060731 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:24.060703    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:33:24.060798 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:24.060763    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:33:24.060798 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:24.060791    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:33:25.030682 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:25.030644    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28" event={"ID":"4ccf3fdf-09b2-48b2-a270-2500fe34b892","Type":"ContainerStarted","Data":"47c918762888c2cd75a3a699e31a44d2e520c34dc1663aea426558abd487af03"}
Oct 30 04:33:25.031131 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:25.030708    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28" event={"ID":"4ccf3fdf-09b2-48b2-a270-2500fe34b892","Type":"ContainerStarted","Data":"8fccff1f6d20d080bb8c69f93343282cebf55836f874d493f6d32a3a2cf88ee8"}
Oct 30 04:33:25.617540 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:33:25.617508    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:33:25.733704 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:25.733675    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:25.734876 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:25.734853    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 04:33:25.734992 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:25.734898    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28" podUID="4ccf3fdf-09b2-48b2-a270-2500fe34b892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 04:33:26.734125 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:26.734093    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 04:33:26.734540 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:26.734162    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28" podUID="4ccf3fdf-09b2-48b2-a270-2500fe34b892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 04:33:27.051977 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.051950    4874 generic.go:356] "Generic (PLEG): container finished" podID="157dc9a9-a3f1-4de0-9787-22207838b7b7" containerID="7d7d7e235b43707b4c5ab8e8aca9330cc6443e91cec5ce978e6599eb16b2696e" exitCode=0
Oct 30 04:33:27.052174 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.052019    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg" event={"ID":"157dc9a9-a3f1-4de0-9787-22207838b7b7","Type":"ContainerDied","Data":"7d7d7e235b43707b4c5ab8e8aca9330cc6443e91cec5ce978e6599eb16b2696e"}
Oct 30 04:33:27.054586 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.054487    4874 generic.go:356] "Generic (PLEG): container finished" podID="908b541f-e807-4403-98c4-41e5cea32837" containerID="3869f7e77e1916369207bd75eb0ef2b7e0db8adb777474de44bfa3fe30828802" exitCode=0
Oct 30 04:33:27.054586 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.054525    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2" event={"ID":"908b541f-e807-4403-98c4-41e5cea32837","Type":"ContainerDied","Data":"3869f7e77e1916369207bd75eb0ef2b7e0db8adb777474de44bfa3fe30828802"}
Oct 30 04:33:27.057437 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.056472    4874 generic.go:356] "Generic (PLEG): container finished" podID="0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1" containerID="8f6812d18fb2347359ebaca20b438759a88954b5e44a28b32681a95b2df8301c" exitCode=0
Oct 30 04:33:27.057437 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.057379    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m" event={"ID":"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1","Type":"ContainerDied","Data":"8f6812d18fb2347359ebaca20b438759a88954b5e44a28b32681a95b2df8301c"}
Oct 30 04:33:27.109455 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.109423    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:33:27.157429 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.157399    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:33:27.159925 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.159907    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:33:27.196668 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.196614    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28" podStartSLOduration=4.196599429 podStartE2EDuration="4.196599429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:33:25.08985703 +0000 UTC m=+2124.502252604" watchObservedRunningTime="2026-10-30 04:33:27.196599429 +0000 UTC m=+2126.608995006"
Oct 30 04:33:27.218678 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.218645    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-workload-socket\") pod \"908b541f-e807-4403-98c4-41e5cea32837\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") "
Oct 30 04:33:27.218871 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.218693    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/908b541f-e807-4403-98c4-41e5cea32837-kube-api-access-q4q87\" (UniqueName: \"kubernetes.io/projected/908b541f-e807-4403-98c4-41e5cea32837-kube-api-access-q4q87\") pod \"908b541f-e807-4403-98c4-41e5cea32837\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") "
Oct 30 04:33:27.218871 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.218725    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-workload-certs\") pod \"157dc9a9-a3f1-4de0-9787-22207838b7b7\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") "
Oct 30 04:33:27.218871 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.218774    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/908b541f-e807-4403-98c4-41e5cea32837-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/908b541f-e807-4403-98c4-41e5cea32837-istio-podinfo\") pod \"908b541f-e807-4403-98c4-41e5cea32837\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") "
Oct 30 04:33:27.218871 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.218844    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/908b541f-e807-4403-98c4-41e5cea32837-istio-token\" (UniqueName: \"kubernetes.io/projected/908b541f-e807-4403-98c4-41e5cea32837-istio-token\") pod \"908b541f-e807-4403-98c4-41e5cea32837\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") "
Oct 30 04:33:27.219131 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.218878    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-data\") pod \"157dc9a9-a3f1-4de0-9787-22207838b7b7\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") "
Oct 30 04:33:27.219131 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.218921    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/157dc9a9-a3f1-4de0-9787-22207838b7b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157dc9a9-a3f1-4de0-9787-22207838b7b7-istiod-ca-cert\") pod \"157dc9a9-a3f1-4de0-9787-22207838b7b7\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") "
Oct 30 04:33:27.219131 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.218962    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-istio-envoy\") pod \"908b541f-e807-4403-98c4-41e5cea32837\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") "
Oct 30 04:33:27.219131 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.219000    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/157dc9a9-a3f1-4de0-9787-22207838b7b7-kube-api-access-5hstc\" (UniqueName: \"kubernetes.io/projected/157dc9a9-a3f1-4de0-9787-22207838b7b7-kube-api-access-5hstc\") pod \"157dc9a9-a3f1-4de0-9787-22207838b7b7\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") "
Oct 30 04:33:27.219131 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.219093    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-token\" (UniqueName: \"kubernetes.io/projected/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-token\") pod \"157dc9a9-a3f1-4de0-9787-22207838b7b7\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") "
Oct 30 04:33:27.219131 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.219126    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-credential-socket\") pod \"157dc9a9-a3f1-4de0-9787-22207838b7b7\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") "
Oct 30 04:33:27.219511 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.219156    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-workload-socket\") pod \"157dc9a9-a3f1-4de0-9787-22207838b7b7\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") "
Oct 30 04:33:27.219511 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.219194    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-credential-socket\") pod \"908b541f-e807-4403-98c4-41e5cea32837\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") "
Oct 30 04:33:27.219511 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.219224    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-workload-certs\") pod \"908b541f-e807-4403-98c4-41e5cea32837\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") "
Oct 30 04:33:27.219511 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.219287    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-podinfo\") pod \"157dc9a9-a3f1-4de0-9787-22207838b7b7\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") "
Oct 30 04:33:27.219511 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.219340    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/908b541f-e807-4403-98c4-41e5cea32837-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/908b541f-e807-4403-98c4-41e5cea32837-istiod-ca-cert\") pod \"908b541f-e807-4403-98c4-41e5cea32837\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") "
Oct 30 04:33:27.219511 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.219409    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-envoy\") pod \"157dc9a9-a3f1-4de0-9787-22207838b7b7\" (UID: \"157dc9a9-a3f1-4de0-9787-22207838b7b7\") "
Oct 30 04:33:27.219511 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.219454    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-istio-data\") pod \"908b541f-e807-4403-98c4-41e5cea32837\" (UID: \"908b541f-e807-4403-98c4-41e5cea32837\") "
Oct 30 04:33:27.220333 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.220178    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-istio-data" pod "908b541f-e807-4403-98c4-41e5cea32837" (UID: "908b541f-e807-4403-98c4-41e5cea32837"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:33:27.220947 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.220920    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-workload-socket" pod "908b541f-e807-4403-98c4-41e5cea32837" (UID: "908b541f-e807-4403-98c4-41e5cea32837"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:33:27.221044 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.220850    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-data" pod "157dc9a9-a3f1-4de0-9787-22207838b7b7" (UID: "157dc9a9-a3f1-4de0-9787-22207838b7b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:33:27.221166 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.221144    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-workload-certs" pod "908b541f-e807-4403-98c4-41e5cea32837" (UID: "908b541f-e807-4403-98c4-41e5cea32837"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:33:27.221778 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.221148    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/157dc9a9-a3f1-4de0-9787-22207838b7b7-istiod-ca-cert" pod "157dc9a9-a3f1-4de0-9787-22207838b7b7" (UID: "157dc9a9-a3f1-4de0-9787-22207838b7b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:33:27.221901 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.221156    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-workload-certs" pod "157dc9a9-a3f1-4de0-9787-22207838b7b7" (UID: "157dc9a9-a3f1-4de0-9787-22207838b7b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:33:27.222162 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.221426    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-workload-socket" pod "157dc9a9-a3f1-4de0-9787-22207838b7b7" (UID: "157dc9a9-a3f1-4de0-9787-22207838b7b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:33:27.222162 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.221464    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-credential-socket" pod "908b541f-e807-4403-98c4-41e5cea32837" (UID: "908b541f-e807-4403-98c4-41e5cea32837"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:33:27.222162 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.221490    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/908b541f-e807-4403-98c4-41e5cea32837-istiod-ca-cert" pod "908b541f-e807-4403-98c4-41e5cea32837" (UID: "908b541f-e807-4403-98c4-41e5cea32837"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:33:27.222162 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.221610    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-credential-socket" pod "157dc9a9-a3f1-4de0-9787-22207838b7b7" (UID: "157dc9a9-a3f1-4de0-9787-22207838b7b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:33:27.223950 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.223909    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908b541f-e807-4403-98c4-41e5cea32837-kube-api-access-q4q87" pod "908b541f-e807-4403-98c4-41e5cea32837" (UID: "908b541f-e807-4403-98c4-41e5cea32837"). InnerVolumeSpecName "kube-api-access-q4q87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:33:27.225012 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.224986    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-podinfo" pod "157dc9a9-a3f1-4de0-9787-22207838b7b7" (UID: "157dc9a9-a3f1-4de0-9787-22207838b7b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:33:27.225314 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.225278    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157dc9a9-a3f1-4de0-9787-22207838b7b7-kube-api-access-5hstc" pod "157dc9a9-a3f1-4de0-9787-22207838b7b7" (UID: "157dc9a9-a3f1-4de0-9787-22207838b7b7"). InnerVolumeSpecName "kube-api-access-5hstc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:33:27.225426 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.225404    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-envoy" pod "157dc9a9-a3f1-4de0-9787-22207838b7b7" (UID: "157dc9a9-a3f1-4de0-9787-22207838b7b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:33:27.225530 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.225507    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-istio-envoy" pod "908b541f-e807-4403-98c4-41e5cea32837" (UID: "908b541f-e807-4403-98c4-41e5cea32837"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:33:27.225625 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.225606    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-token" pod "157dc9a9-a3f1-4de0-9787-22207838b7b7" (UID: "157dc9a9-a3f1-4de0-9787-22207838b7b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:33:27.225899 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.225872    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908b541f-e807-4403-98c4-41e5cea32837-istio-token" pod "908b541f-e807-4403-98c4-41e5cea32837" (UID: "908b541f-e807-4403-98c4-41e5cea32837"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:33:27.226664 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.226644    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/908b541f-e807-4403-98c4-41e5cea32837-istio-podinfo" pod "908b541f-e807-4403-98c4-41e5cea32837" (UID: "908b541f-e807-4403-98c4-41e5cea32837"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:33:27.320299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.320209    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-credential-socket\") pod \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") "
Oct 30 04:33:27.320466 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.320303    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-data\") pod \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") "
Oct 30 04:33:27.320466 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.320334    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-workload-socket\") pod \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") "
Oct 30 04:33:27.320466 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.320378    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-podinfo\") pod \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") "
Oct 30 04:33:27.320466 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.320426    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-envoy\") pod \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") "
Oct 30 04:33:27.320466 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.320455    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-token\" (UniqueName: \"kubernetes.io/projected/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-token\") pod \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") "
Oct 30 04:33:27.320782 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.320469    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-credential-socket" pod "0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1" (UID: "0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:33:27.320782 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.320535    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-kube-api-access-9dvcw\" (UniqueName: \"kubernetes.io/projected/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-kube-api-access-9dvcw\") pod \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") "
Oct 30 04:33:27.320782 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.320548    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-data" pod "0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1" (UID: "0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:33:27.320782 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.320630    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-workload-socket" pod "0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1" (UID: "0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:33:27.320782 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.320687    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istiod-ca-cert\") pod \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") "
Oct 30 04:33:27.320782 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.320762    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-workload-certs\") pod \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\" (UID: \"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1\") "
Oct 30 04:33:27.321130 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.321104    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-workload-certs" pod "0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1" (UID: "0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:33:27.321196 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.321129    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istiod-ca-cert" pod "0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1" (UID: "0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:33:27.321322 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.321304    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/908b541f-e807-4403-98c4-41e5cea32837-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.321384 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.321327    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908b541f-e807-4403-98c4-41e5cea32837-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.321384 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.321342    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.321384 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.321355    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157dc9a9-a3f1-4de0-9787-22207838b7b7-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.321384 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.321369    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.321384 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.321382    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.321657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.321396    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hstc\" (UniqueName: \"kubernetes.io/projected/157dc9a9-a3f1-4de0-9787-22207838b7b7-kube-api-access-5hstc\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.321657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.321412    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.321657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.321427    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.321657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.321440    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.321657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.321455    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.321657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.321468    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.321657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.321483    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.321657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.321499    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.321657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.321512    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.321657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.321523    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.321657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.321536    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.321657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.321550    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/908b541f-e807-4403-98c4-41e5cea32837-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.321657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.321564    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.321657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.321578    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.321657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.321591    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/908b541f-e807-4403-98c4-41e5cea32837-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.321657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.321605    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4q87\" (UniqueName: \"kubernetes.io/projected/908b541f-e807-4403-98c4-41e5cea32837-kube-api-access-q4q87\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.321657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.321620    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157dc9a9-a3f1-4de0-9787-22207838b7b7-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.322737 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.322713    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-kube-api-access-9dvcw" pod "0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1" (UID: "0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1"). InnerVolumeSpecName "kube-api-access-9dvcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:33:27.322737 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.322725    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-token" pod "0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1" (UID: "0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:33:27.322881 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.322748    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-podinfo" pod "0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1" (UID: "0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:33:27.322929 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.322914    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-envoy" pod "0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1" (UID: "0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:33:27.423382 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.423345    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.423382 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.423383    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.423567 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.423399    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dvcw\" (UniqueName: \"kubernetes.io/projected/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-kube-api-access-9dvcw\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.423567 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.423416    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:33:27.734576 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.734542    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 04:33:27.734941 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:27.734597    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28" podUID="4ccf3fdf-09b2-48b2-a270-2500fe34b892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 04:33:28.063451 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.063366    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"
Oct 30 04:33:28.063451 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.063382    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg" event={"ID":"157dc9a9-a3f1-4de0-9787-22207838b7b7","Type":"ContainerDied","Data":"5b5421833629918ca5c2a57d5413280a6fafa0f460014ac2e7135c18dedb24d6"}
Oct 30 04:33:28.063669 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.063444    4874 scope.go:122] "RemoveContainer" containerID="7d7d7e235b43707b4c5ab8e8aca9330cc6443e91cec5ce978e6599eb16b2696e"
Oct 30 04:33:28.078260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.074564    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2" event={"ID":"908b541f-e807-4403-98c4-41e5cea32837","Type":"ContainerDied","Data":"7b5923152c1233164c57c89fc9369e677313ce8d8a1f1b0bea311d010054c4a7"}
Oct 30 04:33:28.078260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.074685    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"
Oct 30 04:33:28.097630 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.097593    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m" event={"ID":"0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1","Type":"ContainerDied","Data":"4ff3d9bd41dacfb0deafb8a780208cb0800fd9c1414c9c85f43406cb1490e1f5"}
Oct 30 04:33:28.097756 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.097604    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"
Oct 30 04:33:28.098594 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.098570    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj"]
Oct 30 04:33:28.099277 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.099255    4874 cpu_manager.go:414] "containerMap: removing container" podUID="908b541f-e807-4403-98c4-41e5cea32837" containerName="istio-proxy"
Oct 30 04:33:28.099349 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.099278    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="908b541f-e807-4403-98c4-41e5cea32837" containerName="istio-proxy"
Oct 30 04:33:28.099349 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.099339    4874 cpu_manager.go:414] "containerMap: removing container" podUID="0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1" containerName="istio-proxy"
Oct 30 04:33:28.099460 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.099349    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1" containerName="istio-proxy"
Oct 30 04:33:28.099460 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.099373    4874 cpu_manager.go:414] "containerMap: removing container" podUID="157dc9a9-a3f1-4de0-9787-22207838b7b7" containerName="istio-proxy"
Oct 30 04:33:28.099460 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.099382    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="157dc9a9-a3f1-4de0-9787-22207838b7b7" containerName="istio-proxy"
Oct 30 04:33:28.099460 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.099398    4874 cpu_manager.go:414] "containerMap: removing container" podUID="1f0cc4fd-bf2b-4642-ab23-e6649237f0fb" containerName="istio-proxy"
Oct 30 04:33:28.099460 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.099406    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f0cc4fd-bf2b-4642-ab23-e6649237f0fb" containerName="istio-proxy"
Oct 30 04:33:28.099605 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.099532    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="908b541f-e807-4403-98c4-41e5cea32837" containerName="istio-proxy"
Oct 30 04:33:28.099605 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.099548    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1" containerName="istio-proxy"
Oct 30 04:33:28.099605 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.099558    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="157dc9a9-a3f1-4de0-9787-22207838b7b7" containerName="istio-proxy"
Oct 30 04:33:28.101832 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.101811    4874 scope.go:122] "RemoveContainer" containerID="3869f7e77e1916369207bd75eb0ef2b7e0db8adb777474de44bfa3fe30828802"
Oct 30 04:33:28.102854 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.102835    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj"
Oct 30 04:33:28.111494 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.111346    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj"]
Oct 30 04:33:28.117802 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.116818    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-86eaa9-serving-cert\""
Oct 30 04:33:28.133911 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.133796    4874 scope.go:122] "RemoveContainer" containerID="8f6812d18fb2347359ebaca20b438759a88954b5e44a28b32681a95b2df8301c"
Oct 30 04:33:28.212846 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.212809    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"]
Oct 30 04:33:28.219036 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.219011    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-czkn2"]
Oct 30 04:33:28.232590 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.232564    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0039db-930d-4c8c-9732-4766e57350a2-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj\" (UID: \"ce0039db-930d-4c8c-9732-4766e57350a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj"
Oct 30 04:33:28.232704 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.232613    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0039db-930d-4c8c-9732-4766e57350a2-metrics-certs\") pod \"maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj\" (UID: \"ce0039db-930d-4c8c-9732-4766e57350a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj"
Oct 30 04:33:28.232750 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.232734    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4qlk\" (UniqueName: \"kubernetes.io/projected/ce0039db-930d-4c8c-9732-4766e57350a2-kube-api-access-w4qlk\") pod \"maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj\" (UID: \"ce0039db-930d-4c8c-9732-4766e57350a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj"
Oct 30 04:33:28.232809 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.232791    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0039db-930d-4c8c-9732-4766e57350a2-maas-api-tls\") pod \"maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj\" (UID: \"ce0039db-930d-4c8c-9732-4766e57350a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj"
Oct 30 04:33:28.251247 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.251218    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"]
Oct 30 04:33:28.257036 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.257016    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-q4qrg"]
Oct 30 04:33:28.301752 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.301721    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"]
Oct 30 04:33:28.321315 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.318464    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lll9m"]
Oct 30 04:33:28.341264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.336070    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0039db-930d-4c8c-9732-4766e57350a2-maas-api-tls\") pod \"maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj\" (UID: \"ce0039db-930d-4c8c-9732-4766e57350a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj"
Oct 30 04:33:28.341264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.336264    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0039db-930d-4c8c-9732-4766e57350a2-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj\" (UID: \"ce0039db-930d-4c8c-9732-4766e57350a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj"
Oct 30 04:33:28.341264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.336306    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0039db-930d-4c8c-9732-4766e57350a2-metrics-certs\") pod \"maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj\" (UID: \"ce0039db-930d-4c8c-9732-4766e57350a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj"
Oct 30 04:33:28.341264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.336392    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4qlk\" (UniqueName: \"kubernetes.io/projected/ce0039db-930d-4c8c-9732-4766e57350a2-kube-api-access-w4qlk\") pod \"maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj\" (UID: \"ce0039db-930d-4c8c-9732-4766e57350a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj"
Oct 30 04:33:28.341264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.337775    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0039db-930d-4c8c-9732-4766e57350a2-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj\" (UID: \"ce0039db-930d-4c8c-9732-4766e57350a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj"
Oct 30 04:33:28.350362 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.350316    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0039db-930d-4c8c-9732-4766e57350a2-metrics-certs\") pod \"maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj\" (UID: \"ce0039db-930d-4c8c-9732-4766e57350a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj"
Oct 30 04:33:28.350574 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.350434    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0039db-930d-4c8c-9732-4766e57350a2-maas-api-tls\") pod \"maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj\" (UID: \"ce0039db-930d-4c8c-9732-4766e57350a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj"
Oct 30 04:33:28.371754 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.371723    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4qlk\" (UniqueName: \"kubernetes.io/projected/ce0039db-930d-4c8c-9732-4766e57350a2-kube-api-access-w4qlk\") pod \"maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj\" (UID: \"ce0039db-930d-4c8c-9732-4766e57350a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj"
Oct 30 04:33:28.434846 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.434805    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj"
Oct 30 04:33:28.730520 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.730490    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1" path="/var/lib/kubelet/pods/0b928ba2-aa86-4b4b-89b4-1d6c8b6b1cc1/volumes"
Oct 30 04:33:28.730964 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.730950    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="157dc9a9-a3f1-4de0-9787-22207838b7b7" path="/var/lib/kubelet/pods/157dc9a9-a3f1-4de0-9787-22207838b7b7/volumes"
Oct 30 04:33:28.731378 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.731364    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="908b541f-e807-4403-98c4-41e5cea32837" path="/var/lib/kubelet/pods/908b541f-e807-4403-98c4-41e5cea32837/volumes"
Oct 30 04:33:28.734518 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.734495    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 04:33:28.734617 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.734533    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28" podUID="4ccf3fdf-09b2-48b2-a270-2500fe34b892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 04:33:28.753950 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:28.753931    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj"]
Oct 30 04:33:28.756777 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:33:28.756744    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0039db_930d_4c8c_9732_4766e57350a2.slice/crio-38bd93e4e93468dbe62aec81ef81da5b9be2a084b4994e518ea3985bada47480 WatchSource:0}: Error finding container 38bd93e4e93468dbe62aec81ef81da5b9be2a084b4994e518ea3985bada47480: Status 404 returned error can't find the container with id 38bd93e4e93468dbe62aec81ef81da5b9be2a084b4994e518ea3985bada47480
Oct 30 04:33:29.107357 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:29.107265    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj" event={"ID":"ce0039db-930d-4c8c-9732-4766e57350a2","Type":"ContainerStarted","Data":"38bd93e4e93468dbe62aec81ef81da5b9be2a084b4994e518ea3985bada47480"}
Oct 30 04:33:29.736791 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:29.735989    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 04:33:29.736791 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:29.736120    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28" podUID="4ccf3fdf-09b2-48b2-a270-2500fe34b892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 04:33:30.115500 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:30.115423    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj" event={"ID":"ce0039db-930d-4c8c-9732-4766e57350a2","Type":"ContainerStarted","Data":"2bd15defe2318ae8c1a0b6c0490f1a5e8d7d7f58c1bc5258e07a4395746dddd2"}
Oct 30 04:33:30.115650 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:30.115551    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj"
Oct 30 04:33:30.195740 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:30.195675    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj" podStartSLOduration=1.740223761 podStartE2EDuration="2.19565994s" totalImagesPullingTime="455.436179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:33:30.193591061 +0000 UTC m=+2129.605986636" watchObservedRunningTime="2026-10-30 04:33:30.19565994 +0000 UTC m=+2129.608055512"
Oct 30 04:33:31.035385 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:31.035353    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:31.035884 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:31.035781    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:31.036590 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:31.036570    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:33:32.069837 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:32.069796    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:33:36.125729 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:36.125700    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj"
Oct 30 04:33:38.474747 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.474708    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"]
Oct 30 04:33:38.478765 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.478743    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.508541 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.508514    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"]
Oct 30 04:33:38.578613 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.578574    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.578834 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.578652    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.578834 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.578736    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.578834 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.578786    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.578978 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.578875    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6cmw\" (UniqueName: \"kubernetes.io/projected/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-kube-api-access-l6cmw\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.579015 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.578995    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.579054 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.579027    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.579107 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.579078    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.579146 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.579102    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.680605 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.680582    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.680748 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.680616    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.680748 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.680649    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6cmw\" (UniqueName: \"kubernetes.io/projected/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-kube-api-access-l6cmw\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.680748 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.680730    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.680913 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.680755    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.680913 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.680781    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.680913 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.680806    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.680913 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.680873    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.681144 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.680968    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.681144 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.681022    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.681144 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.681077    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.681144 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.681131    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.681413 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.681390    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.681614 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.681592    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.683205 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.683181    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.683408 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.683388    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.702697 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.702670    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6cmw\" (UniqueName: \"kubernetes.io/projected/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-kube-api-access-l6cmw\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.704762 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.704745    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:38.790805 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:38.790778    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:39.131816 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:39.131788    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"]
Oct 30 04:33:39.134482 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:33:39.134452    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf6c9aa_0b1b_42ef_acca_f434e0e526c1.slice/crio-5ef2da3bd86e3b45fde79098895a13fc527f1e78e135bc7e151cef402ec4d9db WatchSource:0}: Error finding container 5ef2da3bd86e3b45fde79098895a13fc527f1e78e135bc7e151cef402ec4d9db: Status 404 returned error can't find the container with id 5ef2da3bd86e3b45fde79098895a13fc527f1e78e135bc7e151cef402ec4d9db
Oct 30 04:33:39.136894 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:39.136861    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:33:39.136990 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:39.136940    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:33:39.137054 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:39.136987    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:33:39.166502 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:39.166474    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2" event={"ID":"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1","Type":"ContainerStarted","Data":"5ef2da3bd86e3b45fde79098895a13fc527f1e78e135bc7e151cef402ec4d9db"}
Oct 30 04:33:40.173186 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:40.173144    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2" event={"ID":"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1","Type":"ContainerStarted","Data":"bcd957a378f93c5ac76259854c636380d20bb6b9f8a73a4cde595dad6bef493c"}
Oct 30 04:33:40.245217 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:40.245140    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2" podStartSLOduration=2.245119516 podStartE2EDuration="2.245119516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:33:40.228772876 +0000 UTC m=+2139.641168467" watchObservedRunningTime="2026-10-30 04:33:40.245119516 +0000 UTC m=+2139.657515090"
Oct 30 04:33:40.791604 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:40.791559    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:40.793338 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:40.793310    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 04:33:40.793473 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:40.793358    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2" podUID="eaf6c9aa-0b1b-42ef-acca-f434e0e526c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 04:33:41.791846 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:41.791813    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 04:33:41.792255 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:41.791863    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2" podUID="eaf6c9aa-0b1b-42ef-acca-f434e0e526c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 04:33:42.795687 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:42.795650    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 04:33:42.796324 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:42.796290    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2" podUID="eaf6c9aa-0b1b-42ef-acca-f434e0e526c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 04:33:43.791721 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:43.791675    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 04:33:43.791934 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:43.791742    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2" podUID="eaf6c9aa-0b1b-42ef-acca-f434e0e526c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 04:33:44.792273 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:44.791545    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 04:33:44.792273 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:44.791604    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2" podUID="eaf6c9aa-0b1b-42ef-acca-f434e0e526c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 04:33:45.056304 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.056209    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"]
Oct 30 04:33:45.060341 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.060324    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.100264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.098266    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"]
Oct 30 04:33:45.100963 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.100935    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqkc4\" (UniqueName: \"kubernetes.io/projected/3d59c727-0c93-401b-8f3f-75a568a3bb32-kube-api-access-sqkc4\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.101109 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.101007    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-podinfo\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.101166 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.101112    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-token\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.101204 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.101190    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-envoy\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.101278 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.101226    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-credential-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.101354 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.101337    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-data\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.101417 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.101404    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-workload-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.101484 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.101471    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d59c727-0c93-401b-8f3f-75a568a3bb32-istiod-ca-cert\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.101523 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.101503    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-workload-certs\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.202447 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.202414    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-workload-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.202716 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.202476    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d59c727-0c93-401b-8f3f-75a568a3bb32-istiod-ca-cert\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.202716 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.202517    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-workload-certs\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.202716 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.202560    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqkc4\" (UniqueName: \"kubernetes.io/projected/3d59c727-0c93-401b-8f3f-75a568a3bb32-kube-api-access-sqkc4\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.202716 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.202613    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-podinfo\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.202716 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.202650    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-token\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.202716 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.202713    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-envoy\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.203063 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.202750    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-credential-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.203063 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.202820    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-data\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.203063 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.202855    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-workload-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.203063 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.202909    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-workload-certs\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.203327 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.203231    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-credential-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.203327 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.203300    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-data\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.203567 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.203545    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d59c727-0c93-401b-8f3f-75a568a3bb32-istiod-ca-cert\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.204968 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.204921    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-envoy\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.205071 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.205053    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-podinfo\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.249866 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.249842    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqkc4\" (UniqueName: \"kubernetes.io/projected/3d59c727-0c93-401b-8f3f-75a568a3bb32-kube-api-access-sqkc4\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.253741 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.253722    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-token\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.405178 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.405090    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:45.730660 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.730636    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"]
Oct 30 04:33:45.732438 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:33:45.732411    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d59c727_0c93_401b_8f3f_75a568a3bb32.slice/crio-63e766e75f8770c85595f5964ae5606edb00ef27d90bf9fa00551afbbee6adcd WatchSource:0}: Error finding container 63e766e75f8770c85595f5964ae5606edb00ef27d90bf9fa00551afbbee6adcd: Status 404 returned error can't find the container with id 63e766e75f8770c85595f5964ae5606edb00ef27d90bf9fa00551afbbee6adcd
Oct 30 04:33:45.734998 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.734965    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:33:45.735115 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.735033    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:33:45.735115 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.735064    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:33:45.791058 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.791030    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 04:33:45.791196 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:45.791076    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2" podUID="eaf6c9aa-0b1b-42ef-acca-f434e0e526c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 04:33:46.228139 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:46.228100    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt" event={"ID":"3d59c727-0c93-401b-8f3f-75a568a3bb32","Type":"ContainerStarted","Data":"41031e7e413851942c4d079e097b48397833128987a10690a87752dee8fcdb6f"}
Oct 30 04:33:46.228529 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:46.228157    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt" event={"ID":"3d59c727-0c93-401b-8f3f-75a568a3bb32","Type":"ContainerStarted","Data":"63e766e75f8770c85595f5964ae5606edb00ef27d90bf9fa00551afbbee6adcd"}
Oct 30 04:33:46.295265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:46.292374    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt" podStartSLOduration=1.292350754 podStartE2EDuration="1.292350754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:33:46.283710085 +0000 UTC m=+2145.696105671" watchObservedRunningTime="2026-10-30 04:33:46.292350754 +0000 UTC m=+2145.704746328"
Oct 30 04:33:46.406217 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:46.406168    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:46.407251 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:46.407211    4874 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 30 04:33:46.407337 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:46.407278    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt" podUID="3d59c727-0c93-401b-8f3f-75a568a3bb32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 30 04:33:46.886265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:46.880580    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:46.886265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:46.881946    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:46.891720 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:46.891473    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:33:47.406345 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:47.406301    4874 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 30 04:33:47.406825 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:47.406366    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt" podUID="3d59c727-0c93-401b-8f3f-75a568a3bb32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 30 04:33:48.406468 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:48.406434    4874 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 30 04:33:48.406949 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:48.406491    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt" podUID="3d59c727-0c93-401b-8f3f-75a568a3bb32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 30 04:33:49.405509 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:49.405476    4874 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 30 04:33:49.405726 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:49.405547    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt" podUID="3d59c727-0c93-401b-8f3f-75a568a3bb32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 30 04:33:50.405623 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:50.405587    4874 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 30 04:33:50.406090 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:50.405652    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt" podUID="3d59c727-0c93-401b-8f3f-75a568a3bb32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 30 04:33:51.405575 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:51.405539    4874 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 30 04:33:51.405755 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:51.405593    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt" podUID="3d59c727-0c93-401b-8f3f-75a568a3bb32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 30 04:33:52.779164 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:52.779129    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:52.779993 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:52.779967    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:52.780741 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:52.780722    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:33:55.609445 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:33:55.609406    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:33:58.545718 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:33:58.545687    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:34:08.893076 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:08.893040    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll"]
Oct 30 04:34:13.893629 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:13.893589    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll" podUID="744645d7-7974-4014-b58e-3129785027ad" containerName="payload-pre-processing" containerID="cri-o://a5f21fe0ef49091e862150b4b2bd59cee0e66310435d704387bc780a19324c7e" gracePeriod=25
Oct 30 04:34:15.290348 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:15.290311    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:34:18.844191 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:18.844158    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm"]
Oct 30 04:34:18.844647 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:18.844372    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm" podUID="d8a444bc-289c-49df-a4e0-e85a241e73dc" containerName="maas-api" containerID="cri-o://5aba1f04b2208c8f7933499d998ccb43fa94fc8f41b0707a9b711b83662693c8" gracePeriod=30
Oct 30 04:34:19.008287 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.008250    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz"]
Oct 30 04:34:19.008876 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.008828    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz" podUID="85f9442e-5e24-4a41-bf74-89ffc2cf240a" containerName="payload-processing" containerID="cri-o://e5144aa21f134d6290752e821471830c6772044c7a2dfd38017518007c17e6c0" gracePeriod=30
Oct 30 04:34:19.033807 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.028935    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p"]
Oct 30 04:34:19.033807 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.029139    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p" podUID="6624f77b-35f2-4d06-8a50-87acab5b9c14" containerName="payload-pre-processing" containerID="cri-o://cf77fd36066308fa97e0a4257fea9294fc527d2c1e6b42124369501d54863486" gracePeriod=30
Oct 30 04:34:19.192342 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.191591    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm"
Oct 30 04:34:19.271677 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.271641    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz"]
Oct 30 04:34:19.272059 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.272029    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz" podUID="ef50e538-692c-4669-8861-def5ea47dc1f" containerName="maas-api" containerID="cri-o://1850991bf5dcacbddf09c78c725d8cae960ba34cfd460b5f129d8fac1aabc34e" gracePeriod=30
Oct 30 04:34:19.273923 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.273851    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz"
Oct 30 04:34:19.279336 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.279302    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8a444bc-289c-49df-a4e0-e85a241e73dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a444bc-289c-49df-a4e0-e85a241e73dc-openshift-service-ca\") pod \"d8a444bc-289c-49df-a4e0-e85a241e73dc\" (UID: \"d8a444bc-289c-49df-a4e0-e85a241e73dc\") "
Oct 30 04:34:19.279469 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.279429    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8a444bc-289c-49df-a4e0-e85a241e73dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8a444bc-289c-49df-a4e0-e85a241e73dc-metrics-certs\") pod \"d8a444bc-289c-49df-a4e0-e85a241e73dc\" (UID: \"d8a444bc-289c-49df-a4e0-e85a241e73dc\") "
Oct 30 04:34:19.279821 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.279799    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a444bc-289c-49df-a4e0-e85a241e73dc-kube-api-access-c8s5m\" (UniqueName: \"kubernetes.io/projected/d8a444bc-289c-49df-a4e0-e85a241e73dc-kube-api-access-c8s5m\") pod \"d8a444bc-289c-49df-a4e0-e85a241e73dc\" (UID: \"d8a444bc-289c-49df-a4e0-e85a241e73dc\") "
Oct 30 04:34:19.279922 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.279849    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8a444bc-289c-49df-a4e0-e85a241e73dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a444bc-289c-49df-a4e0-e85a241e73dc-maas-api-tls\") pod \"d8a444bc-289c-49df-a4e0-e85a241e73dc\" (UID: \"d8a444bc-289c-49df-a4e0-e85a241e73dc\") "
Oct 30 04:34:19.280137 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.280108    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a444bc-289c-49df-a4e0-e85a241e73dc-openshift-service-ca" pod "d8a444bc-289c-49df-a4e0-e85a241e73dc" (UID: "d8a444bc-289c-49df-a4e0-e85a241e73dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:34:19.280415 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.280382    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a444bc-289c-49df-a4e0-e85a241e73dc-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:19.285863 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.285831    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a444bc-289c-49df-a4e0-e85a241e73dc-kube-api-access-c8s5m" pod "d8a444bc-289c-49df-a4e0-e85a241e73dc" (UID: "d8a444bc-289c-49df-a4e0-e85a241e73dc"). InnerVolumeSpecName "kube-api-access-c8s5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:34:19.285863 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.285851    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8a444bc-289c-49df-a4e0-e85a241e73dc-maas-api-tls" pod "d8a444bc-289c-49df-a4e0-e85a241e73dc" (UID: "d8a444bc-289c-49df-a4e0-e85a241e73dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:34:19.285983 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.285867    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8a444bc-289c-49df-a4e0-e85a241e73dc-metrics-certs" pod "d8a444bc-289c-49df-a4e0-e85a241e73dc" (UID: "d8a444bc-289c-49df-a4e0-e85a241e73dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:34:19.353194 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.353167    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p"
Oct 30 04:34:19.381636 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.381573    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f9442e-5e24-4a41-bf74-89ffc2cf240a-kube-api-access-pdtr6\" (UniqueName: \"kubernetes.io/projected/85f9442e-5e24-4a41-bf74-89ffc2cf240a-kube-api-access-pdtr6\") pod \"85f9442e-5e24-4a41-bf74-89ffc2cf240a\" (UID: \"85f9442e-5e24-4a41-bf74-89ffc2cf240a\") "
Oct 30 04:34:19.381636 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.381620    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85f9442e-5e24-4a41-bf74-89ffc2cf240a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f9442e-5e24-4a41-bf74-89ffc2cf240a-plugins-config-volume\") pod \"85f9442e-5e24-4a41-bf74-89ffc2cf240a\" (UID: \"85f9442e-5e24-4a41-bf74-89ffc2cf240a\") "
Oct 30 04:34:19.381932 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.381919    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8a444bc-289c-49df-a4e0-e85a241e73dc-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:19.381971 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.381936    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8s5m\" (UniqueName: \"kubernetes.io/projected/d8a444bc-289c-49df-a4e0-e85a241e73dc-kube-api-access-c8s5m\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:19.381971 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.381947    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a444bc-289c-49df-a4e0-e85a241e73dc-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:19.382101 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.382076    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f9442e-5e24-4a41-bf74-89ffc2cf240a-plugins-config-volume" pod "85f9442e-5e24-4a41-bf74-89ffc2cf240a" (UID: "85f9442e-5e24-4a41-bf74-89ffc2cf240a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:34:19.383756 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.383730    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f9442e-5e24-4a41-bf74-89ffc2cf240a-kube-api-access-pdtr6" pod "85f9442e-5e24-4a41-bf74-89ffc2cf240a" (UID: "85f9442e-5e24-4a41-bf74-89ffc2cf240a"). InnerVolumeSpecName "kube-api-access-pdtr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:34:19.482362 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.482329    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6624f77b-35f2-4d06-8a50-87acab5b9c14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6624f77b-35f2-4d06-8a50-87acab5b9c14-plugins-config-volume\") pod \"6624f77b-35f2-4d06-8a50-87acab5b9c14\" (UID: \"6624f77b-35f2-4d06-8a50-87acab5b9c14\") "
Oct 30 04:34:19.482519 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.482376    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6624f77b-35f2-4d06-8a50-87acab5b9c14-kube-api-access-pkdjh\" (UniqueName: \"kubernetes.io/projected/6624f77b-35f2-4d06-8a50-87acab5b9c14-kube-api-access-pkdjh\") pod \"6624f77b-35f2-4d06-8a50-87acab5b9c14\" (UID: \"6624f77b-35f2-4d06-8a50-87acab5b9c14\") "
Oct 30 04:34:19.482781 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.482752    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6624f77b-35f2-4d06-8a50-87acab5b9c14-plugins-config-volume" pod "6624f77b-35f2-4d06-8a50-87acab5b9c14" (UID: "6624f77b-35f2-4d06-8a50-87acab5b9c14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:34:19.482861 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.482843    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6624f77b-35f2-4d06-8a50-87acab5b9c14-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:19.482915 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.482865    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdtr6\" (UniqueName: \"kubernetes.io/projected/85f9442e-5e24-4a41-bf74-89ffc2cf240a-kube-api-access-pdtr6\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:19.482915 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.482879    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f9442e-5e24-4a41-bf74-89ffc2cf240a-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:19.484409 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.484386    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6624f77b-35f2-4d06-8a50-87acab5b9c14-kube-api-access-pkdjh" pod "6624f77b-35f2-4d06-8a50-87acab5b9c14" (UID: "6624f77b-35f2-4d06-8a50-87acab5b9c14"). InnerVolumeSpecName "kube-api-access-pkdjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:34:19.498287 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.498226    4874 generic.go:356] "Generic (PLEG): container finished" podID="85f9442e-5e24-4a41-bf74-89ffc2cf240a" containerID="e5144aa21f134d6290752e821471830c6772044c7a2dfd38017518007c17e6c0" exitCode=0
Oct 30 04:34:19.498287 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.498274    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz" event={"ID":"85f9442e-5e24-4a41-bf74-89ffc2cf240a","Type":"ContainerDied","Data":"e5144aa21f134d6290752e821471830c6772044c7a2dfd38017518007c17e6c0"}
Oct 30 04:34:19.498470 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.498298    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz"
Oct 30 04:34:19.498470 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.498345    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz" event={"ID":"85f9442e-5e24-4a41-bf74-89ffc2cf240a","Type":"ContainerDied","Data":"56c7aff095bceae98899f1e545588ef18f4a5d387e5dbc310b1ff82898be4750"}
Oct 30 04:34:19.498470 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.498378    4874 scope.go:122] "RemoveContainer" containerID="e5144aa21f134d6290752e821471830c6772044c7a2dfd38017518007c17e6c0"
Oct 30 04:34:19.499822 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.499802    4874 generic.go:356] "Generic (PLEG): container finished" podID="d8a444bc-289c-49df-a4e0-e85a241e73dc" containerID="5aba1f04b2208c8f7933499d998ccb43fa94fc8f41b0707a9b711b83662693c8" exitCode=0
Oct 30 04:34:19.499932 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.499862    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm"
Oct 30 04:34:19.499932 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.499876    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm" event={"ID":"d8a444bc-289c-49df-a4e0-e85a241e73dc","Type":"ContainerDied","Data":"5aba1f04b2208c8f7933499d998ccb43fa94fc8f41b0707a9b711b83662693c8"}
Oct 30 04:34:19.499932 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.499905    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm" event={"ID":"d8a444bc-289c-49df-a4e0-e85a241e73dc","Type":"ContainerDied","Data":"f7bd0ad2c49e07d61e8b0e7acd886962c01e87b8a34d021b455872b0db5d37da"}
Oct 30 04:34:19.501394 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.501313    4874 generic.go:356] "Generic (PLEG): container finished" podID="6624f77b-35f2-4d06-8a50-87acab5b9c14" containerID="cf77fd36066308fa97e0a4257fea9294fc527d2c1e6b42124369501d54863486" exitCode=0
Oct 30 04:34:19.501394 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.501356    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p"
Oct 30 04:34:19.501586 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.501398    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p" event={"ID":"6624f77b-35f2-4d06-8a50-87acab5b9c14","Type":"ContainerDied","Data":"cf77fd36066308fa97e0a4257fea9294fc527d2c1e6b42124369501d54863486"}
Oct 30 04:34:19.501586 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.501435    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p" event={"ID":"6624f77b-35f2-4d06-8a50-87acab5b9c14","Type":"ContainerDied","Data":"2b341a9e4671e760943b97174318cd1a51c7cda16d3f18e7c2b949360f3d0f40"}
Oct 30 04:34:19.554789 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.554460    4874 scope.go:122] "RemoveContainer" containerID="e5144aa21f134d6290752e821471830c6772044c7a2dfd38017518007c17e6c0"
Oct 30 04:34:19.572392 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:34:19.555706    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5144aa21f134d6290752e821471830c6772044c7a2dfd38017518007c17e6c0\": container with ID starting with e5144aa21f134d6290752e821471830c6772044c7a2dfd38017518007c17e6c0 not found: ID does not exist" containerID="e5144aa21f134d6290752e821471830c6772044c7a2dfd38017518007c17e6c0"
Oct 30 04:34:19.572392 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.555748    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5144aa21f134d6290752e821471830c6772044c7a2dfd38017518007c17e6c0"} err="failed to get container status \"e5144aa21f134d6290752e821471830c6772044c7a2dfd38017518007c17e6c0\": rpc error: code = NotFound desc = could not find container \"e5144aa21f134d6290752e821471830c6772044c7a2dfd38017518007c17e6c0\": container with ID starting with e5144aa21f134d6290752e821471830c6772044c7a2dfd38017518007c17e6c0 not found: ID does not exist"
Oct 30 04:34:19.572392 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.555775    4874 scope.go:122] "RemoveContainer" containerID="5aba1f04b2208c8f7933499d998ccb43fa94fc8f41b0707a9b711b83662693c8"
Oct 30 04:34:19.595304 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.595271    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkdjh\" (UniqueName: \"kubernetes.io/projected/6624f77b-35f2-4d06-8a50-87acab5b9c14-kube-api-access-pkdjh\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:19.625804 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.625776    4874 scope.go:122] "RemoveContainer" containerID="5aba1f04b2208c8f7933499d998ccb43fa94fc8f41b0707a9b711b83662693c8"
Oct 30 04:34:19.627275 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:34:19.626756    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aba1f04b2208c8f7933499d998ccb43fa94fc8f41b0707a9b711b83662693c8\": container with ID starting with 5aba1f04b2208c8f7933499d998ccb43fa94fc8f41b0707a9b711b83662693c8 not found: ID does not exist" containerID="5aba1f04b2208c8f7933499d998ccb43fa94fc8f41b0707a9b711b83662693c8"
Oct 30 04:34:19.627275 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.626800    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aba1f04b2208c8f7933499d998ccb43fa94fc8f41b0707a9b711b83662693c8"} err="failed to get container status \"5aba1f04b2208c8f7933499d998ccb43fa94fc8f41b0707a9b711b83662693c8\": rpc error: code = NotFound desc = could not find container \"5aba1f04b2208c8f7933499d998ccb43fa94fc8f41b0707a9b711b83662693c8\": container with ID starting with 5aba1f04b2208c8f7933499d998ccb43fa94fc8f41b0707a9b711b83662693c8 not found: ID does not exist"
Oct 30 04:34:19.627275 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.626829    4874 scope.go:122] "RemoveContainer" containerID="cf77fd36066308fa97e0a4257fea9294fc527d2c1e6b42124369501d54863486"
Oct 30 04:34:19.629453 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.629399    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm"]
Oct 30 04:34:19.633026 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.632954    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dbabed-d7b48f5b5-5wjhm"]
Oct 30 04:34:19.661113 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.659406    4874 scope.go:122] "RemoveContainer" containerID="cf77fd36066308fa97e0a4257fea9294fc527d2c1e6b42124369501d54863486"
Oct 30 04:34:19.661113 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:34:19.661046    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf77fd36066308fa97e0a4257fea9294fc527d2c1e6b42124369501d54863486\": container with ID starting with cf77fd36066308fa97e0a4257fea9294fc527d2c1e6b42124369501d54863486 not found: ID does not exist" containerID="cf77fd36066308fa97e0a4257fea9294fc527d2c1e6b42124369501d54863486"
Oct 30 04:34:19.661113 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.661083    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf77fd36066308fa97e0a4257fea9294fc527d2c1e6b42124369501d54863486"} err="failed to get container status \"cf77fd36066308fa97e0a4257fea9294fc527d2c1e6b42124369501d54863486\": rpc error: code = NotFound desc = could not find container \"cf77fd36066308fa97e0a4257fea9294fc527d2c1e6b42124369501d54863486\": container with ID starting with cf77fd36066308fa97e0a4257fea9294fc527d2c1e6b42124369501d54863486 not found: ID does not exist"
Oct 30 04:34:19.682706 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.682637    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz"]
Oct 30 04:34:19.690850 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.690805    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dbabed-7cf5655f64-d72mz"]
Oct 30 04:34:19.756395 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.756346    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p"]
Oct 30 04:34:19.756560 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.756526    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-865cf4767-v4c6p"]
Oct 30 04:34:19.824053 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.824027    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz"
Oct 30 04:34:19.900429 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.900358    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef50e538-692c-4669-8861-def5ea47dc1f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef50e538-692c-4669-8861-def5ea47dc1f-metrics-certs\") pod \"ef50e538-692c-4669-8861-def5ea47dc1f\" (UID: \"ef50e538-692c-4669-8861-def5ea47dc1f\") "
Oct 30 04:34:19.900783 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.900556    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef50e538-692c-4669-8861-def5ea47dc1f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef50e538-692c-4669-8861-def5ea47dc1f-maas-api-tls\") pod \"ef50e538-692c-4669-8861-def5ea47dc1f\" (UID: \"ef50e538-692c-4669-8861-def5ea47dc1f\") "
Oct 30 04:34:19.900783 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.900580    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef50e538-692c-4669-8861-def5ea47dc1f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef50e538-692c-4669-8861-def5ea47dc1f-openshift-service-ca\") pod \"ef50e538-692c-4669-8861-def5ea47dc1f\" (UID: \"ef50e538-692c-4669-8861-def5ea47dc1f\") "
Oct 30 04:34:19.900783 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.900608    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef50e538-692c-4669-8861-def5ea47dc1f-kube-api-access-c8lwq\" (UniqueName: \"kubernetes.io/projected/ef50e538-692c-4669-8861-def5ea47dc1f-kube-api-access-c8lwq\") pod \"ef50e538-692c-4669-8861-def5ea47dc1f\" (UID: \"ef50e538-692c-4669-8861-def5ea47dc1f\") "
Oct 30 04:34:19.901029 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.901007    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef50e538-692c-4669-8861-def5ea47dc1f-openshift-service-ca" pod "ef50e538-692c-4669-8861-def5ea47dc1f" (UID: "ef50e538-692c-4669-8861-def5ea47dc1f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:34:19.903104 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.903072    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef50e538-692c-4669-8861-def5ea47dc1f-metrics-certs" pod "ef50e538-692c-4669-8861-def5ea47dc1f" (UID: "ef50e538-692c-4669-8861-def5ea47dc1f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:34:19.903203 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.903127    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef50e538-692c-4669-8861-def5ea47dc1f-maas-api-tls" pod "ef50e538-692c-4669-8861-def5ea47dc1f" (UID: "ef50e538-692c-4669-8861-def5ea47dc1f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:34:19.903203 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:19.903172    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef50e538-692c-4669-8861-def5ea47dc1f-kube-api-access-c8lwq" pod "ef50e538-692c-4669-8861-def5ea47dc1f" (UID: "ef50e538-692c-4669-8861-def5ea47dc1f"). InnerVolumeSpecName "kube-api-access-c8lwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:34:20.002315 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:20.002278    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef50e538-692c-4669-8861-def5ea47dc1f-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:20.002315 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:20.002307    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef50e538-692c-4669-8861-def5ea47dc1f-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:20.002315 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:20.002320    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8lwq\" (UniqueName: \"kubernetes.io/projected/ef50e538-692c-4669-8861-def5ea47dc1f-kube-api-access-c8lwq\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:20.002526 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:20.002333    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef50e538-692c-4669-8861-def5ea47dc1f-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:20.507835 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:20.507803    4874 generic.go:356] "Generic (PLEG): container finished" podID="ef50e538-692c-4669-8861-def5ea47dc1f" containerID="1850991bf5dcacbddf09c78c725d8cae960ba34cfd460b5f129d8fac1aabc34e" exitCode=0
Oct 30 04:34:20.508045 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:20.507857    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz"
Oct 30 04:34:20.508045 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:20.507879    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz" event={"ID":"ef50e538-692c-4669-8861-def5ea47dc1f","Type":"ContainerDied","Data":"1850991bf5dcacbddf09c78c725d8cae960ba34cfd460b5f129d8fac1aabc34e"}
Oct 30 04:34:20.508045 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:20.507934    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz" event={"ID":"ef50e538-692c-4669-8861-def5ea47dc1f","Type":"ContainerDied","Data":"faffcb13b7c865c79692054da368c42e8e6d7064878c4678b159692f4dec6323"}
Oct 30 04:34:20.508045 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:20.507967    4874 scope.go:122] "RemoveContainer" containerID="1850991bf5dcacbddf09c78c725d8cae960ba34cfd460b5f129d8fac1aabc34e"
Oct 30 04:34:20.518649 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:20.518627    4874 scope.go:122] "RemoveContainer" containerID="1850991bf5dcacbddf09c78c725d8cae960ba34cfd460b5f129d8fac1aabc34e"
Oct 30 04:34:20.518919 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:34:20.518895    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1850991bf5dcacbddf09c78c725d8cae960ba34cfd460b5f129d8fac1aabc34e\": container with ID starting with 1850991bf5dcacbddf09c78c725d8cae960ba34cfd460b5f129d8fac1aabc34e not found: ID does not exist" containerID="1850991bf5dcacbddf09c78c725d8cae960ba34cfd460b5f129d8fac1aabc34e"
Oct 30 04:34:20.518993 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:20.518928    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1850991bf5dcacbddf09c78c725d8cae960ba34cfd460b5f129d8fac1aabc34e"} err="failed to get container status \"1850991bf5dcacbddf09c78c725d8cae960ba34cfd460b5f129d8fac1aabc34e\": rpc error: code = NotFound desc = could not find container \"1850991bf5dcacbddf09c78c725d8cae960ba34cfd460b5f129d8fac1aabc34e\": container with ID starting with 1850991bf5dcacbddf09c78c725d8cae960ba34cfd460b5f129d8fac1aabc34e not found: ID does not exist"
Oct 30 04:34:20.556573 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:20.556550    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz"]
Oct 30 04:34:20.567896 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:20.567877    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-153a44-5d6c47db6b-cd8jz"]
Oct 30 04:34:20.729347 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:20.729315    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6624f77b-35f2-4d06-8a50-87acab5b9c14" path="/var/lib/kubelet/pods/6624f77b-35f2-4d06-8a50-87acab5b9c14/volumes"
Oct 30 04:34:20.729734 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:20.729715    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f9442e-5e24-4a41-bf74-89ffc2cf240a" path="/var/lib/kubelet/pods/85f9442e-5e24-4a41-bf74-89ffc2cf240a/volumes"
Oct 30 04:34:20.730097 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:20.730082    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8a444bc-289c-49df-a4e0-e85a241e73dc" path="/var/lib/kubelet/pods/d8a444bc-289c-49df-a4e0-e85a241e73dc/volumes"
Oct 30 04:34:20.730819 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:20.730796    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef50e538-692c-4669-8861-def5ea47dc1f" path="/var/lib/kubelet/pods/ef50e538-692c-4669-8861-def5ea47dc1f/volumes"
Oct 30 04:34:21.337397 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:21.337344    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll" podUID="744645d7-7974-4014-b58e-3129785027ad" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:34:22.328018 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:22.327988    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"]
Oct 30 04:34:22.328208 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:22.328183    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s" podUID="6a69deca-d8cf-4ea4-bb1f-15bee6a789a4" containerName="istio-proxy" containerID="cri-o://26fd51259d1c410f7211180d96dac6bc4614dcb74b50bb50e7fabaa855b90f14" gracePeriod=30
Oct 30 04:34:23.180217 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:23.180180    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"]
Oct 30 04:34:23.180700 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:23.180465    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp" podUID="4ded57b2-1fa2-414f-9f3b-b598eee59dad" containerName="istio-proxy" containerID="cri-o://f2462aa95d441531f3f480b76676f133e50032e57234e0f5f8183ab6a807ee2e" gracePeriod=30
Oct 30 04:34:23.791758 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:23.791723    4874 patch_prober.go:29] interesting pod/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 04:34:23.791928 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:23.791785    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s" podUID="6a69deca-d8cf-4ea4-bb1f-15bee6a789a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:34:25.088294 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:25.088258    4874 patch_prober.go:29] interesting pod/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 04:34:25.088673 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:25.088322    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp" podUID="4ded57b2-1fa2-414f-9f3b-b598eee59dad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:34:25.609447 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:34:25.609405    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:34:27.553215 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.553178    4874 generic.go:356] "Generic (PLEG): container finished" podID="6a69deca-d8cf-4ea4-bb1f-15bee6a789a4" containerID="26fd51259d1c410f7211180d96dac6bc4614dcb74b50bb50e7fabaa855b90f14" exitCode=0
Oct 30 04:34:27.553617 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.553269    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s" event={"ID":"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4","Type":"ContainerDied","Data":"26fd51259d1c410f7211180d96dac6bc4614dcb74b50bb50e7fabaa855b90f14"}
Oct 30 04:34:27.553617 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.553358    4874 scope.go:122] "RemoveContainer" containerID="220354a581ef639ab086606c6b464cc89579d421dae2d836bc820fd532dc51a8"
Oct 30 04:34:27.591026 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.591001    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:34:27.702891 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.702819    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-credential-socket\") pod \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") "
Oct 30 04:34:27.703024 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.702891    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istiod-ca-cert\") pod \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") "
Oct 30 04:34:27.703024 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.702928    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-kube-api-access-wc662\" (UniqueName: \"kubernetes.io/projected/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-kube-api-access-wc662\") pod \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") "
Oct 30 04:34:27.703024 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.703007    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-podinfo\") pod \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") "
Oct 30 04:34:27.703134 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.703104    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-credential-socket" pod "6a69deca-d8cf-4ea4-bb1f-15bee6a789a4" (UID: "6a69deca-d8cf-4ea4-bb1f-15bee6a789a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:34:27.703215 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.703189    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-envoy\") pod \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") "
Oct 30 04:34:27.703317 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.703298    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-token\" (UniqueName: \"kubernetes.io/projected/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-token\") pod \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") "
Oct 30 04:34:27.703390 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.703311    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istiod-ca-cert" pod "6a69deca-d8cf-4ea4-bb1f-15bee6a789a4" (UID: "6a69deca-d8cf-4ea4-bb1f-15bee6a789a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:34:27.703390 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.703335    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-workload-certs\") pod \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") "
Oct 30 04:34:27.703390 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.703375    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-workload-socket\") pod \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") "
Oct 30 04:34:27.703561 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.703406    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-data\") pod \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\" (UID: \"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4\") "
Oct 30 04:34:27.703672 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.703650    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-workload-certs" pod "6a69deca-d8cf-4ea4-bb1f-15bee6a789a4" (UID: "6a69deca-d8cf-4ea4-bb1f-15bee6a789a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:34:27.703734 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.703668    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-workload-socket" pod "6a69deca-d8cf-4ea4-bb1f-15bee6a789a4" (UID: "6a69deca-d8cf-4ea4-bb1f-15bee6a789a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:34:27.703909 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.703853    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-data" pod "6a69deca-d8cf-4ea4-bb1f-15bee6a789a4" (UID: "6a69deca-d8cf-4ea4-bb1f-15bee6a789a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:34:27.703998 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.703975    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:27.704062 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.703996    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:27.704062 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.704012    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:27.704062 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.704023    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:27.704062 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.704037    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:27.705335 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.705314    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-token" pod "6a69deca-d8cf-4ea4-bb1f-15bee6a789a4" (UID: "6a69deca-d8cf-4ea4-bb1f-15bee6a789a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:34:27.705439 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.705397    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-envoy" pod "6a69deca-d8cf-4ea4-bb1f-15bee6a789a4" (UID: "6a69deca-d8cf-4ea4-bb1f-15bee6a789a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:34:27.705603 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.705588    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-kube-api-access-wc662" pod "6a69deca-d8cf-4ea4-bb1f-15bee6a789a4" (UID: "6a69deca-d8cf-4ea4-bb1f-15bee6a789a4"). InnerVolumeSpecName "kube-api-access-wc662". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:34:27.705668 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.705604    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-podinfo" pod "6a69deca-d8cf-4ea4-bb1f-15bee6a789a4" (UID: "6a69deca-d8cf-4ea4-bb1f-15bee6a789a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:34:27.806708 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.806666    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:27.806708 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.806710    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:27.806978 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.806728    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc662\" (UniqueName: \"kubernetes.io/projected/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-kube-api-access-wc662\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:27.806978 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:27.806747    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:28.431615 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.431587    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:34:28.512982 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.512952    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-credential-socket\") pod \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") "
Oct 30 04:34:28.513148 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.512993    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-token\" (UniqueName: \"kubernetes.io/projected/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-token\") pod \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") "
Oct 30 04:34:28.513148 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.513032    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istiod-ca-cert\") pod \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") "
Oct 30 04:34:28.513148 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.513113    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-envoy\") pod \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") "
Oct 30 04:34:28.513148 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.513135    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-workload-socket\") pod \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") "
Oct 30 04:34:28.513408 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.513153    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-workload-certs\") pod \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") "
Oct 30 04:34:28.513408 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.513264    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-credential-socket" pod "4ded57b2-1fa2-414f-9f3b-b598eee59dad" (UID: "4ded57b2-1fa2-414f-9f3b-b598eee59dad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:34:28.513502 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.513466    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-podinfo\") pod \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") "
Oct 30 04:34:28.513502 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.513486    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-workload-certs" pod "4ded57b2-1fa2-414f-9f3b-b598eee59dad" (UID: "4ded57b2-1fa2-414f-9f3b-b598eee59dad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:34:28.513619 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.513517    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-workload-socket" pod "4ded57b2-1fa2-414f-9f3b-b598eee59dad" (UID: "4ded57b2-1fa2-414f-9f3b-b598eee59dad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:34:28.513619 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.513525    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ded57b2-1fa2-414f-9f3b-b598eee59dad-kube-api-access-8wlr6\" (UniqueName: \"kubernetes.io/projected/4ded57b2-1fa2-414f-9f3b-b598eee59dad-kube-api-access-8wlr6\") pod \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") "
Oct 30 04:34:28.513619 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.513564    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istiod-ca-cert" pod "4ded57b2-1fa2-414f-9f3b-b598eee59dad" (UID: "4ded57b2-1fa2-414f-9f3b-b598eee59dad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:34:28.513776 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.513618    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-data\") pod \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\" (UID: \"4ded57b2-1fa2-414f-9f3b-b598eee59dad\") "
Oct 30 04:34:28.514065 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.513995    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-data" pod "4ded57b2-1fa2-414f-9f3b-b598eee59dad" (UID: "4ded57b2-1fa2-414f-9f3b-b598eee59dad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:34:28.514190 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.514122    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:28.514190 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.514140    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:28.514190 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.514153    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:28.514190 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.514166    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:28.514190 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.514179    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:28.516033 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.516009    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-envoy" pod "4ded57b2-1fa2-414f-9f3b-b598eee59dad" (UID: "4ded57b2-1fa2-414f-9f3b-b598eee59dad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:34:28.516146 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.516049    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ded57b2-1fa2-414f-9f3b-b598eee59dad-kube-api-access-8wlr6" pod "4ded57b2-1fa2-414f-9f3b-b598eee59dad" (UID: "4ded57b2-1fa2-414f-9f3b-b598eee59dad"). InnerVolumeSpecName "kube-api-access-8wlr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:34:28.516146 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.516129    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-token" pod "4ded57b2-1fa2-414f-9f3b-b598eee59dad" (UID: "4ded57b2-1fa2-414f-9f3b-b598eee59dad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:34:28.516146 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.516138    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-podinfo" pod "4ded57b2-1fa2-414f-9f3b-b598eee59dad" (UID: "4ded57b2-1fa2-414f-9f3b-b598eee59dad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:34:28.560396 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.560369    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"
Oct 30 04:34:28.560803 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.560399    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s" event={"ID":"6a69deca-d8cf-4ea4-bb1f-15bee6a789a4","Type":"ContainerDied","Data":"06014e139cbf0dad704200631fc5f52f4bbb15d729f008c4950720e2895ccf10"}
Oct 30 04:34:28.560803 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.560454    4874 scope.go:122] "RemoveContainer" containerID="26fd51259d1c410f7211180d96dac6bc4614dcb74b50bb50e7fabaa855b90f14"
Oct 30 04:34:28.562227 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.562195    4874 generic.go:356] "Generic (PLEG): container finished" podID="4ded57b2-1fa2-414f-9f3b-b598eee59dad" containerID="f2462aa95d441531f3f480b76676f133e50032e57234e0f5f8183ab6a807ee2e" exitCode=0
Oct 30 04:34:28.562337 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.562286    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"
Oct 30 04:34:28.562380 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.562287    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp" event={"ID":"4ded57b2-1fa2-414f-9f3b-b598eee59dad","Type":"ContainerDied","Data":"f2462aa95d441531f3f480b76676f133e50032e57234e0f5f8183ab6a807ee2e"}
Oct 30 04:34:28.562419 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.562405    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp" event={"ID":"4ded57b2-1fa2-414f-9f3b-b598eee59dad","Type":"ContainerDied","Data":"e75aea795baf42b63466af7684c022d6ecf28004fba0a80b956c7949a7e8d8f9"}
Oct 30 04:34:28.572267 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.572234    4874 scope.go:122] "RemoveContainer" containerID="f2462aa95d441531f3f480b76676f133e50032e57234e0f5f8183ab6a807ee2e"
Oct 30 04:34:28.581028 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.581013    4874 scope.go:122] "RemoveContainer" containerID="04e566564f84cf9972937d9e23690af8b7698c68c76bb00f9d86c254a07bef16"
Oct 30 04:34:28.588857 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.588842    4874 scope.go:122] "RemoveContainer" containerID="f2462aa95d441531f3f480b76676f133e50032e57234e0f5f8183ab6a807ee2e"
Oct 30 04:34:28.589092 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:34:28.589071    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2462aa95d441531f3f480b76676f133e50032e57234e0f5f8183ab6a807ee2e\": container with ID starting with f2462aa95d441531f3f480b76676f133e50032e57234e0f5f8183ab6a807ee2e not found: ID does not exist" containerID="f2462aa95d441531f3f480b76676f133e50032e57234e0f5f8183ab6a807ee2e"
Oct 30 04:34:28.589151 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.589100    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2462aa95d441531f3f480b76676f133e50032e57234e0f5f8183ab6a807ee2e"} err="failed to get container status \"f2462aa95d441531f3f480b76676f133e50032e57234e0f5f8183ab6a807ee2e\": rpc error: code = NotFound desc = could not find container \"f2462aa95d441531f3f480b76676f133e50032e57234e0f5f8183ab6a807ee2e\": container with ID starting with f2462aa95d441531f3f480b76676f133e50032e57234e0f5f8183ab6a807ee2e not found: ID does not exist"
Oct 30 04:34:28.589151 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.589118    4874 scope.go:122] "RemoveContainer" containerID="04e566564f84cf9972937d9e23690af8b7698c68c76bb00f9d86c254a07bef16"
Oct 30 04:34:28.589413 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:34:28.589394    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04e566564f84cf9972937d9e23690af8b7698c68c76bb00f9d86c254a07bef16\": container with ID starting with 04e566564f84cf9972937d9e23690af8b7698c68c76bb00f9d86c254a07bef16 not found: ID does not exist" containerID="04e566564f84cf9972937d9e23690af8b7698c68c76bb00f9d86c254a07bef16"
Oct 30 04:34:28.589461 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.589418    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04e566564f84cf9972937d9e23690af8b7698c68c76bb00f9d86c254a07bef16"} err="failed to get container status \"04e566564f84cf9972937d9e23690af8b7698c68c76bb00f9d86c254a07bef16\": rpc error: code = NotFound desc = could not find container \"04e566564f84cf9972937d9e23690af8b7698c68c76bb00f9d86c254a07bef16\": container with ID starting with 04e566564f84cf9972937d9e23690af8b7698c68c76bb00f9d86c254a07bef16 not found: ID does not exist"
Oct 30 04:34:28.611065 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.611043    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"]
Oct 30 04:34:28.615207 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.615187    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wlr6\" (UniqueName: \"kubernetes.io/projected/4ded57b2-1fa2-414f-9f3b-b598eee59dad-kube-api-access-8wlr6\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:28.615301 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.615211    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:28.615301 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.615226    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:28.615301 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.615261    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ded57b2-1fa2-414f-9f3b-b598eee59dad-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:28.615908 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.615889    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-dbabed-openshift-default-7cddf558b9-4n99s"]
Oct 30 04:34:28.649164 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.649144    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"]
Oct 30 04:34:28.656658 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.656640    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-153a44-openshift-default-57c49f9997-jxwzp"]
Oct 30 04:34:28.728945 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.728920    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ded57b2-1fa2-414f-9f3b-b598eee59dad" path="/var/lib/kubelet/pods/4ded57b2-1fa2-414f-9f3b-b598eee59dad/volumes"
Oct 30 04:34:28.729374 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:28.729361    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a69deca-d8cf-4ea4-bb1f-15bee6a789a4" path="/var/lib/kubelet/pods/6a69deca-d8cf-4ea4-bb1f-15bee6a789a4/volumes"
Oct 30 04:34:31.318272 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:31.318205    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll" podUID="744645d7-7974-4014-b58e-3129785027ad" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:34:34.164145 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:34.164089    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll"
Oct 30 04:34:34.240255 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:34.240215    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/744645d7-7974-4014-b58e-3129785027ad-kube-api-access-6vnw5\" (UniqueName: \"kubernetes.io/projected/744645d7-7974-4014-b58e-3129785027ad-kube-api-access-6vnw5\") pod \"744645d7-7974-4014-b58e-3129785027ad\" (UID: \"744645d7-7974-4014-b58e-3129785027ad\") "
Oct 30 04:34:34.240410 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:34.240279    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/744645d7-7974-4014-b58e-3129785027ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744645d7-7974-4014-b58e-3129785027ad-plugins-config-volume\") pod \"744645d7-7974-4014-b58e-3129785027ad\" (UID: \"744645d7-7974-4014-b58e-3129785027ad\") "
Oct 30 04:34:34.240641 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:34.240613    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744645d7-7974-4014-b58e-3129785027ad-plugins-config-volume" pod "744645d7-7974-4014-b58e-3129785027ad" (UID: "744645d7-7974-4014-b58e-3129785027ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:34:34.242191 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:34.242169    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744645d7-7974-4014-b58e-3129785027ad-kube-api-access-6vnw5" pod "744645d7-7974-4014-b58e-3129785027ad" (UID: "744645d7-7974-4014-b58e-3129785027ad"). InnerVolumeSpecName "kube-api-access-6vnw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:34:34.341681 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:34.341652    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vnw5\" (UniqueName: \"kubernetes.io/projected/744645d7-7974-4014-b58e-3129785027ad-kube-api-access-6vnw5\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:34.341681 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:34.341678    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744645d7-7974-4014-b58e-3129785027ad-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:34.600941 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:34.600909    4874 generic.go:356] "Generic (PLEG): container finished" podID="744645d7-7974-4014-b58e-3129785027ad" containerID="a5f21fe0ef49091e862150b4b2bd59cee0e66310435d704387bc780a19324c7e" exitCode=0
Oct 30 04:34:34.601118 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:34.600952    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll"
Oct 30 04:34:34.601118 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:34.600958    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll" event={"ID":"744645d7-7974-4014-b58e-3129785027ad","Type":"ContainerDied","Data":"a5f21fe0ef49091e862150b4b2bd59cee0e66310435d704387bc780a19324c7e"}
Oct 30 04:34:34.601118 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:34.601008    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll" event={"ID":"744645d7-7974-4014-b58e-3129785027ad","Type":"ContainerDied","Data":"0cb8306c6382598828d74c3ed84f5b2eb7cf11d56ed94ccd05619a7e5b8b8164"}
Oct 30 04:34:34.601118 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:34.601031    4874 scope.go:122] "RemoveContainer" containerID="a5f21fe0ef49091e862150b4b2bd59cee0e66310435d704387bc780a19324c7e"
Oct 30 04:34:34.610624 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:34.610606    4874 scope.go:122] "RemoveContainer" containerID="a5f21fe0ef49091e862150b4b2bd59cee0e66310435d704387bc780a19324c7e"
Oct 30 04:34:34.610866 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:34:34.610846    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5f21fe0ef49091e862150b4b2bd59cee0e66310435d704387bc780a19324c7e\": container with ID starting with a5f21fe0ef49091e862150b4b2bd59cee0e66310435d704387bc780a19324c7e not found: ID does not exist" containerID="a5f21fe0ef49091e862150b4b2bd59cee0e66310435d704387bc780a19324c7e"
Oct 30 04:34:34.610932 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:34.610873    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5f21fe0ef49091e862150b4b2bd59cee0e66310435d704387bc780a19324c7e"} err="failed to get container status \"a5f21fe0ef49091e862150b4b2bd59cee0e66310435d704387bc780a19324c7e\": rpc error: code = NotFound desc = could not find container \"a5f21fe0ef49091e862150b4b2bd59cee0e66310435d704387bc780a19324c7e\": container with ID starting with a5f21fe0ef49091e862150b4b2bd59cee0e66310435d704387bc780a19324c7e not found: ID does not exist"
Oct 30 04:34:34.649903 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:34.649879    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll"]
Oct 30 04:34:34.653923 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:34.653901    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-153a44-7d899665f9-zl5ll"]
Oct 30 04:34:34.728760 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:34.728737    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="744645d7-7974-4014-b58e-3129785027ad" path="/var/lib/kubelet/pods/744645d7-7974-4014-b58e-3129785027ad/volumes"
Oct 30 04:34:41.642113 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:41.642079    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd"]
Oct 30 04:34:41.642635 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:41.642382    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd" podUID="80a534d3-0927-4aa0-882a-c5734776b387" containerName="maas-api" containerID="cri-o://020e19d403e5f4148d0bb8e8fa23f12aa62fe6f3e925c17807e5fbd0dcd2d064" gracePeriod=30
Oct 30 04:34:41.929589 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:41.928903    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd"
Oct 30 04:34:42.030425 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:42.030395    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80a534d3-0927-4aa0-882a-c5734776b387-kube-api-access-92p9g\" (UniqueName: \"kubernetes.io/projected/80a534d3-0927-4aa0-882a-c5734776b387-kube-api-access-92p9g\") pod \"80a534d3-0927-4aa0-882a-c5734776b387\" (UID: \"80a534d3-0927-4aa0-882a-c5734776b387\") "
Oct 30 04:34:42.030585 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:42.030452    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80a534d3-0927-4aa0-882a-c5734776b387-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a534d3-0927-4aa0-882a-c5734776b387-metrics-certs\") pod \"80a534d3-0927-4aa0-882a-c5734776b387\" (UID: \"80a534d3-0927-4aa0-882a-c5734776b387\") "
Oct 30 04:34:42.030585 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:42.030495    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80a534d3-0927-4aa0-882a-c5734776b387-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a534d3-0927-4aa0-882a-c5734776b387-openshift-service-ca\") pod \"80a534d3-0927-4aa0-882a-c5734776b387\" (UID: \"80a534d3-0927-4aa0-882a-c5734776b387\") "
Oct 30 04:34:42.030585 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:42.030529    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80a534d3-0927-4aa0-882a-c5734776b387-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80a534d3-0927-4aa0-882a-c5734776b387-maas-api-tls\") pod \"80a534d3-0927-4aa0-882a-c5734776b387\" (UID: \"80a534d3-0927-4aa0-882a-c5734776b387\") "
Oct 30 04:34:42.030838 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:42.030812    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80a534d3-0927-4aa0-882a-c5734776b387-openshift-service-ca" pod "80a534d3-0927-4aa0-882a-c5734776b387" (UID: "80a534d3-0927-4aa0-882a-c5734776b387"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:34:42.030968 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:42.030952    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a534d3-0927-4aa0-882a-c5734776b387-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:42.032572 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:42.032552    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a534d3-0927-4aa0-882a-c5734776b387-kube-api-access-92p9g" pod "80a534d3-0927-4aa0-882a-c5734776b387" (UID: "80a534d3-0927-4aa0-882a-c5734776b387"). InnerVolumeSpecName "kube-api-access-92p9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:34:42.033043 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:42.033019    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80a534d3-0927-4aa0-882a-c5734776b387-metrics-certs" pod "80a534d3-0927-4aa0-882a-c5734776b387" (UID: "80a534d3-0927-4aa0-882a-c5734776b387"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:34:42.033111 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:42.033029    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80a534d3-0927-4aa0-882a-c5734776b387-maas-api-tls" pod "80a534d3-0927-4aa0-882a-c5734776b387" (UID: "80a534d3-0927-4aa0-882a-c5734776b387"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:34:42.135067 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:42.135026    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92p9g\" (UniqueName: \"kubernetes.io/projected/80a534d3-0927-4aa0-882a-c5734776b387-kube-api-access-92p9g\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:42.135067 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:42.135068    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a534d3-0927-4aa0-882a-c5734776b387-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:42.135382 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:42.135083    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80a534d3-0927-4aa0-882a-c5734776b387-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:42.645783 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:42.645743    4874 generic.go:356] "Generic (PLEG): container finished" podID="80a534d3-0927-4aa0-882a-c5734776b387" containerID="020e19d403e5f4148d0bb8e8fa23f12aa62fe6f3e925c17807e5fbd0dcd2d064" exitCode=0
Oct 30 04:34:42.646303 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:42.645792    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd"
Oct 30 04:34:42.646303 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:42.645824    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd" event={"ID":"80a534d3-0927-4aa0-882a-c5734776b387","Type":"ContainerDied","Data":"020e19d403e5f4148d0bb8e8fa23f12aa62fe6f3e925c17807e5fbd0dcd2d064"}
Oct 30 04:34:42.646303 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:42.645876    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd" event={"ID":"80a534d3-0927-4aa0-882a-c5734776b387","Type":"ContainerDied","Data":"f33a065d4c4c8e717c73ede3093eaf9a4b759cb1fc92732ccaa003fdb47df2ca"}
Oct 30 04:34:42.646303 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:42.645901    4874 scope.go:122] "RemoveContainer" containerID="020e19d403e5f4148d0bb8e8fa23f12aa62fe6f3e925c17807e5fbd0dcd2d064"
Oct 30 04:34:42.658016 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:42.657925    4874 scope.go:122] "RemoveContainer" containerID="020e19d403e5f4148d0bb8e8fa23f12aa62fe6f3e925c17807e5fbd0dcd2d064"
Oct 30 04:34:42.658383 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:34:42.658333    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"020e19d403e5f4148d0bb8e8fa23f12aa62fe6f3e925c17807e5fbd0dcd2d064\": container with ID starting with 020e19d403e5f4148d0bb8e8fa23f12aa62fe6f3e925c17807e5fbd0dcd2d064 not found: ID does not exist" containerID="020e19d403e5f4148d0bb8e8fa23f12aa62fe6f3e925c17807e5fbd0dcd2d064"
Oct 30 04:34:42.658383 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:42.658372    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"020e19d403e5f4148d0bb8e8fa23f12aa62fe6f3e925c17807e5fbd0dcd2d064"} err="failed to get container status \"020e19d403e5f4148d0bb8e8fa23f12aa62fe6f3e925c17807e5fbd0dcd2d064\": rpc error: code = NotFound desc = could not find container \"020e19d403e5f4148d0bb8e8fa23f12aa62fe6f3e925c17807e5fbd0dcd2d064\": container with ID starting with 020e19d403e5f4148d0bb8e8fa23f12aa62fe6f3e925c17807e5fbd0dcd2d064 not found: ID does not exist"
Oct 30 04:34:42.693856 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:42.693832    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd"]
Oct 30 04:34:42.705781 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:42.705760    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-448a3a-57f4b4b556-mwvjd"]
Oct 30 04:34:42.729080 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:42.729049    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80a534d3-0927-4aa0-882a-c5734776b387" path="/var/lib/kubelet/pods/80a534d3-0927-4aa0-882a-c5734776b387/volumes"
Oct 30 04:34:46.351336 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:46.351297    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:34:52.477527 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:52.477494    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj"]
Oct 30 04:34:52.478071 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:52.477728    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj" podUID="ce0039db-930d-4c8c-9732-4766e57350a2" containerName="maas-api" containerID="cri-o://2bd15defe2318ae8c1a0b6c0490f1a5e8d7d7f58c1bc5258e07a4395746dddd2" gracePeriod=30
Oct 30 04:34:52.706504 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:52.706472    4874 generic.go:356] "Generic (PLEG): container finished" podID="ce0039db-930d-4c8c-9732-4766e57350a2" containerID="2bd15defe2318ae8c1a0b6c0490f1a5e8d7d7f58c1bc5258e07a4395746dddd2" exitCode=0
Oct 30 04:34:52.706656 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:52.706517    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj" event={"ID":"ce0039db-930d-4c8c-9732-4766e57350a2","Type":"ContainerDied","Data":"2bd15defe2318ae8c1a0b6c0490f1a5e8d7d7f58c1bc5258e07a4395746dddd2"}
Oct 30 04:34:52.782576 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:52.782306    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj"
Oct 30 04:34:52.872487 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:52.872455    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce0039db-930d-4c8c-9732-4766e57350a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0039db-930d-4c8c-9732-4766e57350a2-metrics-certs\") pod \"ce0039db-930d-4c8c-9732-4766e57350a2\" (UID: \"ce0039db-930d-4c8c-9732-4766e57350a2\") "
Oct 30 04:34:52.872677 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:52.872502    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce0039db-930d-4c8c-9732-4766e57350a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0039db-930d-4c8c-9732-4766e57350a2-openshift-service-ca\") pod \"ce0039db-930d-4c8c-9732-4766e57350a2\" (UID: \"ce0039db-930d-4c8c-9732-4766e57350a2\") "
Oct 30 04:34:52.872677 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:52.872547    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce0039db-930d-4c8c-9732-4766e57350a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0039db-930d-4c8c-9732-4766e57350a2-maas-api-tls\") pod \"ce0039db-930d-4c8c-9732-4766e57350a2\" (UID: \"ce0039db-930d-4c8c-9732-4766e57350a2\") "
Oct 30 04:34:52.872677 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:52.872651    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce0039db-930d-4c8c-9732-4766e57350a2-kube-api-access-w4qlk\" (UniqueName: \"kubernetes.io/projected/ce0039db-930d-4c8c-9732-4766e57350a2-kube-api-access-w4qlk\") pod \"ce0039db-930d-4c8c-9732-4766e57350a2\" (UID: \"ce0039db-930d-4c8c-9732-4766e57350a2\") "
Oct 30 04:34:52.872923 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:52.872886    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce0039db-930d-4c8c-9732-4766e57350a2-openshift-service-ca" pod "ce0039db-930d-4c8c-9732-4766e57350a2" (UID: "ce0039db-930d-4c8c-9732-4766e57350a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:34:52.873076 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:52.873059    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0039db-930d-4c8c-9732-4766e57350a2-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:52.875074 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:52.875045    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce0039db-930d-4c8c-9732-4766e57350a2-metrics-certs" pod "ce0039db-930d-4c8c-9732-4766e57350a2" (UID: "ce0039db-930d-4c8c-9732-4766e57350a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:34:52.875169 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:52.875072    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce0039db-930d-4c8c-9732-4766e57350a2-kube-api-access-w4qlk" pod "ce0039db-930d-4c8c-9732-4766e57350a2" (UID: "ce0039db-930d-4c8c-9732-4766e57350a2"). InnerVolumeSpecName "kube-api-access-w4qlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:34:52.875169 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:52.875072    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce0039db-930d-4c8c-9732-4766e57350a2-maas-api-tls" pod "ce0039db-930d-4c8c-9732-4766e57350a2" (UID: "ce0039db-930d-4c8c-9732-4766e57350a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:34:52.981135 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:52.980995    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0039db-930d-4c8c-9732-4766e57350a2-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:52.981135 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:52.981036    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4qlk\" (UniqueName: \"kubernetes.io/projected/ce0039db-930d-4c8c-9732-4766e57350a2-kube-api-access-w4qlk\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:52.981135 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:52.981053    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0039db-930d-4c8c-9732-4766e57350a2-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:53.636140 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:53.636105    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"]
Oct 30 04:34:53.636590 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:53.636371    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx" podUID="3f12d804-4aab-4672-af70-0f668cbb63a9" containerName="istio-proxy" containerID="cri-o://80ca450ce9bb30d6b506ca828ef04dcaa352b05230f4a46572afee9b3eae1ce7" gracePeriod=30
Oct 30 04:34:53.714461 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:53.714431    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj"
Oct 30 04:34:53.714624 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:53.714433    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj" event={"ID":"ce0039db-930d-4c8c-9732-4766e57350a2","Type":"ContainerDied","Data":"38bd93e4e93468dbe62aec81ef81da5b9be2a084b4994e518ea3985bada47480"}
Oct 30 04:34:53.714624 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:53.714563    4874 scope.go:122] "RemoveContainer" containerID="2bd15defe2318ae8c1a0b6c0490f1a5e8d7d7f58c1bc5258e07a4395746dddd2"
Oct 30 04:34:53.815444 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:53.815400    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj"]
Oct 30 04:34:53.826059 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:53.826032    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-86eaa9-54864c99b6-fwrcj"]
Oct 30 04:34:54.727827 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:54.727794    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce0039db-930d-4c8c-9732-4766e57350a2" path="/var/lib/kubelet/pods/ce0039db-930d-4c8c-9732-4766e57350a2/volumes"
Oct 30 04:34:55.609821 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:34:55.609786    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:34:58.926881 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:58.926857    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:34:59.053687 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.053654    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f12d804-4aab-4672-af70-0f668cbb63a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f12d804-4aab-4672-af70-0f668cbb63a9-istiod-ca-cert\") pod \"3f12d804-4aab-4672-af70-0f668cbb63a9\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") "
Oct 30 04:34:59.053848 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.053731    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-envoy\") pod \"3f12d804-4aab-4672-af70-0f668cbb63a9\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") "
Oct 30 04:34:59.053848 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.053779    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-token\" (UniqueName: \"kubernetes.io/projected/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-token\") pod \"3f12d804-4aab-4672-af70-0f668cbb63a9\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") "
Oct 30 04:34:59.053848 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.053800    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-data\") pod \"3f12d804-4aab-4672-af70-0f668cbb63a9\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") "
Oct 30 04:34:59.053848 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.053846    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-workload-certs\") pod \"3f12d804-4aab-4672-af70-0f668cbb63a9\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") "
Oct 30 04:34:59.054082 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.053866    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-workload-socket\") pod \"3f12d804-4aab-4672-af70-0f668cbb63a9\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") "
Oct 30 04:34:59.054082 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.053883    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-credential-socket\") pod \"3f12d804-4aab-4672-af70-0f668cbb63a9\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") "
Oct 30 04:34:59.054082 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.053937    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f12d804-4aab-4672-af70-0f668cbb63a9-kube-api-access-kt76g\" (UniqueName: \"kubernetes.io/projected/3f12d804-4aab-4672-af70-0f668cbb63a9-kube-api-access-kt76g\") pod \"3f12d804-4aab-4672-af70-0f668cbb63a9\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") "
Oct 30 04:34:59.054082 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.053988    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-podinfo\") pod \"3f12d804-4aab-4672-af70-0f668cbb63a9\" (UID: \"3f12d804-4aab-4672-af70-0f668cbb63a9\") "
Oct 30 04:34:59.054082 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.054036    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f12d804-4aab-4672-af70-0f668cbb63a9-istiod-ca-cert" pod "3f12d804-4aab-4672-af70-0f668cbb63a9" (UID: "3f12d804-4aab-4672-af70-0f668cbb63a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:34:59.054384 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.054174    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-workload-certs" pod "3f12d804-4aab-4672-af70-0f668cbb63a9" (UID: "3f12d804-4aab-4672-af70-0f668cbb63a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:34:59.054384 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.054186    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-workload-socket" pod "3f12d804-4aab-4672-af70-0f668cbb63a9" (UID: "3f12d804-4aab-4672-af70-0f668cbb63a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:34:59.054384 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.054214    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-data" pod "3f12d804-4aab-4672-af70-0f668cbb63a9" (UID: "3f12d804-4aab-4672-af70-0f668cbb63a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:34:59.054553 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.054398    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-credential-socket" pod "3f12d804-4aab-4672-af70-0f668cbb63a9" (UID: "3f12d804-4aab-4672-af70-0f668cbb63a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:34:59.054553 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.054503    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f12d804-4aab-4672-af70-0f668cbb63a9-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:59.054553 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.054527    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:59.054553 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.054541    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:59.054764 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.054553    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:59.054764 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.054567    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:59.056123 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.056095    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-envoy" pod "3f12d804-4aab-4672-af70-0f668cbb63a9" (UID: "3f12d804-4aab-4672-af70-0f668cbb63a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:34:59.056224 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.056134    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-podinfo" pod "3f12d804-4aab-4672-af70-0f668cbb63a9" (UID: "3f12d804-4aab-4672-af70-0f668cbb63a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:34:59.056443 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.056422    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-token" pod "3f12d804-4aab-4672-af70-0f668cbb63a9" (UID: "3f12d804-4aab-4672-af70-0f668cbb63a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:34:59.056528 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.056499    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f12d804-4aab-4672-af70-0f668cbb63a9-kube-api-access-kt76g" pod "3f12d804-4aab-4672-af70-0f668cbb63a9" (UID: "3f12d804-4aab-4672-af70-0f668cbb63a9"). InnerVolumeSpecName "kube-api-access-kt76g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:34:59.155225 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.155174    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt76g\" (UniqueName: \"kubernetes.io/projected/3f12d804-4aab-4672-af70-0f668cbb63a9-kube-api-access-kt76g\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:59.155225 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.155197    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:59.155225 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.155207    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:59.155225 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.155216    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f12d804-4aab-4672-af70-0f668cbb63a9-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:34:59.759530 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.759491    4874 generic.go:356] "Generic (PLEG): container finished" podID="3f12d804-4aab-4672-af70-0f668cbb63a9" containerID="80ca450ce9bb30d6b506ca828ef04dcaa352b05230f4a46572afee9b3eae1ce7" exitCode=0
Oct 30 04:34:59.759682 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.759538    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"
Oct 30 04:34:59.759682 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.759579    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx" event={"ID":"3f12d804-4aab-4672-af70-0f668cbb63a9","Type":"ContainerDied","Data":"80ca450ce9bb30d6b506ca828ef04dcaa352b05230f4a46572afee9b3eae1ce7"}
Oct 30 04:34:59.759682 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.759632    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx" event={"ID":"3f12d804-4aab-4672-af70-0f668cbb63a9","Type":"ContainerDied","Data":"8fa4332237630378eb36caa7b670ad0c97c998121da4f2db12200bb5d5299e2d"}
Oct 30 04:34:59.759682 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.759663    4874 scope.go:122] "RemoveContainer" containerID="80ca450ce9bb30d6b506ca828ef04dcaa352b05230f4a46572afee9b3eae1ce7"
Oct 30 04:34:59.769633 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.769617    4874 scope.go:122] "RemoveContainer" containerID="bc48517e00eb1827516ad79a3321d8299a415c407134f7128bd98fcc68a7c69f"
Oct 30 04:34:59.777866 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.777848    4874 scope.go:122] "RemoveContainer" containerID="80ca450ce9bb30d6b506ca828ef04dcaa352b05230f4a46572afee9b3eae1ce7"
Oct 30 04:34:59.778130 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:34:59.778110    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80ca450ce9bb30d6b506ca828ef04dcaa352b05230f4a46572afee9b3eae1ce7\": container with ID starting with 80ca450ce9bb30d6b506ca828ef04dcaa352b05230f4a46572afee9b3eae1ce7 not found: ID does not exist" containerID="80ca450ce9bb30d6b506ca828ef04dcaa352b05230f4a46572afee9b3eae1ce7"
Oct 30 04:34:59.778201 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.778143    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80ca450ce9bb30d6b506ca828ef04dcaa352b05230f4a46572afee9b3eae1ce7"} err="failed to get container status \"80ca450ce9bb30d6b506ca828ef04dcaa352b05230f4a46572afee9b3eae1ce7\": rpc error: code = NotFound desc = could not find container \"80ca450ce9bb30d6b506ca828ef04dcaa352b05230f4a46572afee9b3eae1ce7\": container with ID starting with 80ca450ce9bb30d6b506ca828ef04dcaa352b05230f4a46572afee9b3eae1ce7 not found: ID does not exist"
Oct 30 04:34:59.778201 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.778169    4874 scope.go:122] "RemoveContainer" containerID="bc48517e00eb1827516ad79a3321d8299a415c407134f7128bd98fcc68a7c69f"
Oct 30 04:34:59.778447 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:34:59.778430    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc48517e00eb1827516ad79a3321d8299a415c407134f7128bd98fcc68a7c69f\": container with ID starting with bc48517e00eb1827516ad79a3321d8299a415c407134f7128bd98fcc68a7c69f not found: ID does not exist" containerID="bc48517e00eb1827516ad79a3321d8299a415c407134f7128bd98fcc68a7c69f"
Oct 30 04:34:59.778496 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.778453    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc48517e00eb1827516ad79a3321d8299a415c407134f7128bd98fcc68a7c69f"} err="failed to get container status \"bc48517e00eb1827516ad79a3321d8299a415c407134f7128bd98fcc68a7c69f\": rpc error: code = NotFound desc = could not find container \"bc48517e00eb1827516ad79a3321d8299a415c407134f7128bd98fcc68a7c69f\": container with ID starting with bc48517e00eb1827516ad79a3321d8299a415c407134f7128bd98fcc68a7c69f not found: ID does not exist"
Oct 30 04:34:59.826198 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.826173    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"]
Oct 30 04:34:59.835673 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:34:59.835644    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-448a3a-openshift-default-6cc5f9c945-7l8zx"]
Oct 30 04:35:00.729536 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:00.729507    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f12d804-4aab-4672-af70-0f668cbb63a9" path="/var/lib/kubelet/pods/3f12d804-4aab-4672-af70-0f668cbb63a9/volumes"
Oct 30 04:35:01.322836 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:01.322803    4874 scope.go:122] "RemoveContainer" containerID="5dc6f642eba3e0558d05f1776048ea8e35508b28f2724a9cfdc7733be767989d"
Oct 30 04:35:05.037908 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:05.037871    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:35:10.721267 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.721219    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"]
Oct 30 04:35:10.721816 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.721801    4874 cpu_manager.go:414] "containerMap: removing container" podUID="6624f77b-35f2-4d06-8a50-87acab5b9c14" containerName="payload-pre-processing"
Oct 30 04:35:10.721873 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.721819    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6624f77b-35f2-4d06-8a50-87acab5b9c14" containerName="payload-pre-processing"
Oct 30 04:35:10.721873 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.721842    4874 cpu_manager.go:414] "containerMap: removing container" podUID="744645d7-7974-4014-b58e-3129785027ad" containerName="payload-pre-processing"
Oct 30 04:35:10.721873 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.721850    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="744645d7-7974-4014-b58e-3129785027ad" containerName="payload-pre-processing"
Oct 30 04:35:10.721971 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.721875    4874 cpu_manager.go:414] "containerMap: removing container" podUID="ce0039db-930d-4c8c-9732-4766e57350a2" containerName="maas-api"
Oct 30 04:35:10.721971 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.721884    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce0039db-930d-4c8c-9732-4766e57350a2" containerName="maas-api"
Oct 30 04:35:10.721971 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.721895    4874 cpu_manager.go:414] "containerMap: removing container" podUID="6a69deca-d8cf-4ea4-bb1f-15bee6a789a4" containerName="istio-proxy"
Oct 30 04:35:10.721971 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.721902    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a69deca-d8cf-4ea4-bb1f-15bee6a789a4" containerName="istio-proxy"
Oct 30 04:35:10.721971 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.721912    4874 cpu_manager.go:414] "containerMap: removing container" podUID="4ded57b2-1fa2-414f-9f3b-b598eee59dad" containerName="istio-proxy"
Oct 30 04:35:10.721971 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.721918    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ded57b2-1fa2-414f-9f3b-b598eee59dad" containerName="istio-proxy"
Oct 30 04:35:10.721971 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.721929    4874 cpu_manager.go:414] "containerMap: removing container" podUID="85f9442e-5e24-4a41-bf74-89ffc2cf240a" containerName="payload-processing"
Oct 30 04:35:10.721971 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.721937    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f9442e-5e24-4a41-bf74-89ffc2cf240a" containerName="payload-processing"
Oct 30 04:35:10.721971 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.721947    4874 cpu_manager.go:414] "containerMap: removing container" podUID="80a534d3-0927-4aa0-882a-c5734776b387" containerName="maas-api"
Oct 30 04:35:10.721971 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.721955    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80a534d3-0927-4aa0-882a-c5734776b387" containerName="maas-api"
Oct 30 04:35:10.722281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.721978    4874 cpu_manager.go:414] "containerMap: removing container" podUID="4ded57b2-1fa2-414f-9f3b-b598eee59dad" containerName="istio-proxy"
Oct 30 04:35:10.722281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.721985    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ded57b2-1fa2-414f-9f3b-b598eee59dad" containerName="istio-proxy"
Oct 30 04:35:10.722281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.722002    4874 cpu_manager.go:414] "containerMap: removing container" podUID="3f12d804-4aab-4672-af70-0f668cbb63a9" containerName="istio-proxy"
Oct 30 04:35:10.722281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.722009    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f12d804-4aab-4672-af70-0f668cbb63a9" containerName="istio-proxy"
Oct 30 04:35:10.722281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.722018    4874 cpu_manager.go:414] "containerMap: removing container" podUID="3f12d804-4aab-4672-af70-0f668cbb63a9" containerName="istio-proxy"
Oct 30 04:35:10.722281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.722025    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f12d804-4aab-4672-af70-0f668cbb63a9" containerName="istio-proxy"
Oct 30 04:35:10.722281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.722033    4874 cpu_manager.go:414] "containerMap: removing container" podUID="d8a444bc-289c-49df-a4e0-e85a241e73dc" containerName="maas-api"
Oct 30 04:35:10.722281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.722040    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a444bc-289c-49df-a4e0-e85a241e73dc" containerName="maas-api"
Oct 30 04:35:10.722281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.722050    4874 cpu_manager.go:414] "containerMap: removing container" podUID="ef50e538-692c-4669-8861-def5ea47dc1f" containerName="maas-api"
Oct 30 04:35:10.722281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.722058    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef50e538-692c-4669-8861-def5ea47dc1f" containerName="maas-api"
Oct 30 04:35:10.722281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.722069    4874 cpu_manager.go:414] "containerMap: removing container" podUID="6a69deca-d8cf-4ea4-bb1f-15bee6a789a4" containerName="istio-proxy"
Oct 30 04:35:10.722281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.722077    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a69deca-d8cf-4ea4-bb1f-15bee6a789a4" containerName="istio-proxy"
Oct 30 04:35:10.722281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.722160    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="6624f77b-35f2-4d06-8a50-87acab5b9c14" containerName="payload-pre-processing"
Oct 30 04:35:10.722281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.722170    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="744645d7-7974-4014-b58e-3129785027ad" containerName="payload-pre-processing"
Oct 30 04:35:10.722281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.722183    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce0039db-930d-4c8c-9732-4766e57350a2" containerName="maas-api"
Oct 30 04:35:10.722281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.722191    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a69deca-d8cf-4ea4-bb1f-15bee6a789a4" containerName="istio-proxy"
Oct 30 04:35:10.722281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.722198    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ded57b2-1fa2-414f-9f3b-b598eee59dad" containerName="istio-proxy"
Oct 30 04:35:10.722281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.722206    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f9442e-5e24-4a41-bf74-89ffc2cf240a" containerName="payload-processing"
Oct 30 04:35:10.722281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.722214    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="80a534d3-0927-4aa0-882a-c5734776b387" containerName="maas-api"
Oct 30 04:35:10.722281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.722225    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f12d804-4aab-4672-af70-0f668cbb63a9" containerName="istio-proxy"
Oct 30 04:35:10.722281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.722234    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f12d804-4aab-4672-af70-0f668cbb63a9" containerName="istio-proxy"
Oct 30 04:35:10.722281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.722256    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a444bc-289c-49df-a4e0-e85a241e73dc" containerName="maas-api"
Oct 30 04:35:10.722281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.722273    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef50e538-692c-4669-8861-def5ea47dc1f" containerName="maas-api"
Oct 30 04:35:10.727692 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.727667    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.741804 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.741781    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-a2f984-openshift-default-dockercfg-pqhlj\""
Oct 30 04:35:10.757169 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.757137    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"]
Oct 30 04:35:10.870616 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.870587    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-istio-data\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.870784 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.870628    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b488e852-f9e1-49ea-ac86-b3f409299806-istio-token\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.870784 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.870648    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-workload-socket\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.870784 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.870772    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnqqs\" (UniqueName: \"kubernetes.io/projected/b488e852-f9e1-49ea-ac86-b3f409299806-kube-api-access-fnqqs\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.870922 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.870808    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-istio-envoy\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.870922 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.870856    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-workload-certs\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.870922 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.870886    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b488e852-f9e1-49ea-ac86-b3f409299806-istiod-ca-cert\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.871026 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.870937    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-credential-socket\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.871026 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.871016    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b488e852-f9e1-49ea-ac86-b3f409299806-istio-podinfo\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.972285 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.972186    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b488e852-f9e1-49ea-ac86-b3f409299806-istio-token\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.972285 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.972221    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-workload-socket\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.972285 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.972269    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnqqs\" (UniqueName: \"kubernetes.io/projected/b488e852-f9e1-49ea-ac86-b3f409299806-kube-api-access-fnqqs\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.972558 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.972305    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-istio-envoy\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.972558 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.972324    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-workload-certs\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.972558 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.972343    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b488e852-f9e1-49ea-ac86-b3f409299806-istiod-ca-cert\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.972558 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.972382    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-credential-socket\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.972558 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.972486    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b488e852-f9e1-49ea-ac86-b3f409299806-istio-podinfo\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.972558 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.972526    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-istio-data\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.972878 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.972698    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-workload-socket\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.972878 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.972794    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-workload-certs\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.972878 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.972847    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-credential-socket\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.973036 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.972902    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-istio-data\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.973036 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.973005    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b488e852-f9e1-49ea-ac86-b3f409299806-istiod-ca-cert\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.974594 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.974564    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-istio-envoy\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.974842 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.974827    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b488e852-f9e1-49ea-ac86-b3f409299806-istio-podinfo\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.995633 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.995605    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnqqs\" (UniqueName: \"kubernetes.io/projected/b488e852-f9e1-49ea-ac86-b3f409299806-kube-api-access-fnqqs\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:10.995826 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:10.995810    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b488e852-f9e1-49ea-ac86-b3f409299806-istio-token\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:11.041510 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:11.041488    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:11.362739 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:35:11.362703    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb488e852_f9e1_49ea_ac86_b3f409299806.slice/crio-ffcaf87313b4a7ea4d01ffb802c0ba0d4b3cf1d0bf1d60c2e60fb73ae786ff83 WatchSource:0}: Error finding container ffcaf87313b4a7ea4d01ffb802c0ba0d4b3cf1d0bf1d60c2e60fb73ae786ff83: Status 404 returned error can't find the container with id ffcaf87313b4a7ea4d01ffb802c0ba0d4b3cf1d0bf1d60c2e60fb73ae786ff83
Oct 30 04:35:11.365293 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:11.365234    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:35:11.365395 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:11.365325    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:35:11.365395 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:11.365357    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:35:11.365395 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:11.365362    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"]
Oct 30 04:35:11.830143 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:11.830101    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc" event={"ID":"b488e852-f9e1-49ea-ac86-b3f409299806","Type":"ContainerStarted","Data":"b91fd174552fbc3629dea60259187237f2ff95ff8f678ddb4b0fdff2775455cb"}
Oct 30 04:35:11.830891 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:11.830858    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc" event={"ID":"b488e852-f9e1-49ea-ac86-b3f409299806","Type":"ContainerStarted","Data":"ffcaf87313b4a7ea4d01ffb802c0ba0d4b3cf1d0bf1d60c2e60fb73ae786ff83"}
Oct 30 04:35:11.907089 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:11.907016    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc" podStartSLOduration=1.9069933369999998 podStartE2EDuration="1.906993337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:35:11.88845477 +0000 UTC m=+2231.300850413" watchObservedRunningTime="2026-10-30 04:35:11.906993337 +0000 UTC m=+2231.319388916"
Oct 30 04:35:12.042383 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:12.042332    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:12.043873 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:12.043843    4874 patch_prober.go:29] interesting pod/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 04:35:12.043991 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:12.043902    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc" podUID="b488e852-f9e1-49ea-ac86-b3f409299806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 04:35:13.041793 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:13.041757    4874 patch_prober.go:29] interesting pod/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 04:35:13.042187 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:13.041820    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc" podUID="b488e852-f9e1-49ea-ac86-b3f409299806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 04:35:14.041961 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:14.041929    4874 patch_prober.go:29] interesting pod/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 04:35:14.042377 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:14.041988    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc" podUID="b488e852-f9e1-49ea-ac86-b3f409299806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 04:35:15.046568 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:15.046528    4874 patch_prober.go:29] interesting pod/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 04:35:15.047153 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:15.047120    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc" podUID="b488e852-f9e1-49ea-ac86-b3f409299806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 04:35:16.042157 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:16.042120    4874 patch_prober.go:29] interesting pod/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 04:35:16.042371 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:16.042176    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc" podUID="b488e852-f9e1-49ea-ac86-b3f409299806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 04:35:17.042428 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:17.042394    4874 patch_prober.go:29] interesting pod/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 04:35:17.042798 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:17.042445    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc" podUID="b488e852-f9e1-49ea-ac86-b3f409299806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 04:35:18.275973 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:18.275937    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:18.276485 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:18.276384    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:18.277309 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:18.277286    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:35:25.608739 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:35:25.608706    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:35:30.041959 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:30.041921    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:35:45.653495 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:45.653462    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:35:53.300589 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.300555    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"]
Oct 30 04:35:53.301065 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.301049    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a69deca-d8cf-4ea4-bb1f-15bee6a789a4" containerName="istio-proxy"
Oct 30 04:35:53.301109 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.301067    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ded57b2-1fa2-414f-9f3b-b598eee59dad" containerName="istio-proxy"
Oct 30 04:35:53.304544 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.304523    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.318523 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.318501    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-5d74b9-openshift-default-dockercfg-d8777\""
Oct 30 04:35:53.332620 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.332593    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"]
Oct 30 04:35:53.358671 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.358641    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.358804 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.358680    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.358804 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.358702    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.358882 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.358834    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.358922 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.358892    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.358955 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.358921    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.358955 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.358948    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.359100 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.359088    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.359137 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.359110    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdvxw\" (UniqueName: \"kubernetes.io/projected/e31f5bf2-ef53-40df-a264-32d5bfb39dff-kube-api-access-kdvxw\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.460151 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.460116    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.460151 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.460157    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.460401 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.460211    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.460401 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.460271    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.460401 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.460300    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.460401 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.460338    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.460628 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.460420    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.460628 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.460447    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdvxw\" (UniqueName: \"kubernetes.io/projected/e31f5bf2-ef53-40df-a264-32d5bfb39dff-kube-api-access-kdvxw\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.460628 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.460483    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.460799 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.460619    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.460799 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.460641    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.460913 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.460831    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.460913 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.460877    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.461123 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.461099    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.462708 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.462685    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.462916 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.462897    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.482598 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.482575    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.482698 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.482658    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdvxw\" (UniqueName: \"kubernetes.io/projected/e31f5bf2-ef53-40df-a264-32d5bfb39dff-kube-api-access-kdvxw\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.618794 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.618706    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:53.938515 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.938479    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"]
Oct 30 04:35:53.940686 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:35:53.940659    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode31f5bf2_ef53_40df_a264_32d5bfb39dff.slice/crio-ea4819458818c22e2930a04834aab735ce348b8316a5f13b02a92f74069644d9 WatchSource:0}: Error finding container ea4819458818c22e2930a04834aab735ce348b8316a5f13b02a92f74069644d9: Status 404 returned error can't find the container with id ea4819458818c22e2930a04834aab735ce348b8316a5f13b02a92f74069644d9
Oct 30 04:35:53.942860 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.942829    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:35:53.942968 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.942912    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:35:53.942968 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:53.942955    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:35:54.062044 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:54.062012    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq" event={"ID":"e31f5bf2-ef53-40df-a264-32d5bfb39dff","Type":"ContainerStarted","Data":"479c26424bbe4b7ad25fe4aa037ba902aa834bc69029ae909e138e642e77d328"}
Oct 30 04:35:54.062230 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:54.062063    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq" event={"ID":"e31f5bf2-ef53-40df-a264-32d5bfb39dff","Type":"ContainerStarted","Data":"ea4819458818c22e2930a04834aab735ce348b8316a5f13b02a92f74069644d9"}
Oct 30 04:35:54.106462 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:54.106350    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq" podStartSLOduration=1.106335113 podStartE2EDuration="1.106335113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:35:54.105385053 +0000 UTC m=+2273.517780625" watchObservedRunningTime="2026-10-30 04:35:54.106335113 +0000 UTC m=+2273.518730688"
Oct 30 04:35:54.619153 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:54.619110    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:35:54.620862 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:54.620828    4874 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 04:35:54.621008 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:54.620889    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq" podUID="e31f5bf2-ef53-40df-a264-32d5bfb39dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 04:35:55.612990 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:35:55.612952    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:35:55.619458 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:55.619436    4874 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 04:35:55.619772 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:55.619492    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq" podUID="e31f5bf2-ef53-40df-a264-32d5bfb39dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 04:35:56.620096 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:56.620061    4874 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 04:35:56.620493 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:56.620116    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq" podUID="e31f5bf2-ef53-40df-a264-32d5bfb39dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 04:35:57.619222 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:57.619189    4874 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 04:35:57.619495 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:57.619267    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq" podUID="e31f5bf2-ef53-40df-a264-32d5bfb39dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 04:35:58.619085 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:58.619049    4874 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 04:35:58.619550 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:58.619102    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq" podUID="e31f5bf2-ef53-40df-a264-32d5bfb39dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 04:35:59.619811 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:59.619773    4874 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 04:35:59.620207 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:35:59.619833    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq" podUID="e31f5bf2-ef53-40df-a264-32d5bfb39dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 04:36:00.869114 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:00.869077    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:36:00.869972 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:00.869944    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:36:00.872130 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:00.872111    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:36:08.343598 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.343553    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"]
Oct 30 04:36:08.347203 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.347186    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.372017 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.371992    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"]
Oct 30 04:36:08.436276 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.436222    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.436443 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.436286    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.436443 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.436317    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.436443 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.436388    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.436443 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.436436    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.436676 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.436466    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dabf1f5-a781-45e8-a9f2-622109ab647b-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.436676 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.436581    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.436676 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.436610    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnsb2\" (UniqueName: \"kubernetes.io/projected/7dabf1f5-a781-45e8-a9f2-622109ab647b-kube-api-access-xnsb2\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.436676 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.436648    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.537771 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.537741    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.537771 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.537774    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.537998 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.537792    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.537998 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.537816    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.537998 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.537843    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.537998 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.537873    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dabf1f5-a781-45e8-a9f2-622109ab647b-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.537998 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.537951    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.537998 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.537978    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnsb2\" (UniqueName: \"kubernetes.io/projected/7dabf1f5-a781-45e8-a9f2-622109ab647b-kube-api-access-xnsb2\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.538396 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.538017    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.538396 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.538207    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.538396 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.538380    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.538396 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.538388    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.538569 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.538434    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.538820 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.538795    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dabf1f5-a781-45e8-a9f2-622109ab647b-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.540212 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.540187    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.540364 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.540345    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.560320 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.560296    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.561080 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.561057    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnsb2\" (UniqueName: \"kubernetes.io/projected/7dabf1f5-a781-45e8-a9f2-622109ab647b-kube-api-access-xnsb2\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.660426 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.660358    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:08.980416 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.980392    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"]
Oct 30 04:36:08.982089 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:36:08.982061    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dabf1f5_a781_45e8_a9f2_622109ab647b.slice/crio-e06b9ae26e61c517f3fcb06971df6208afe882a068f8e7e50e67782c5a78617b WatchSource:0}: Error finding container e06b9ae26e61c517f3fcb06971df6208afe882a068f8e7e50e67782c5a78617b: Status 404 returned error can't find the container with id e06b9ae26e61c517f3fcb06971df6208afe882a068f8e7e50e67782c5a78617b
Oct 30 04:36:08.984536 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.984505    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:36:08.984614 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.984567    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:36:08.984614 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:08.984596    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:36:09.182616 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:09.182570    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89" event={"ID":"7dabf1f5-a781-45e8-a9f2-622109ab647b","Type":"ContainerStarted","Data":"40b8ecfa1e6591f9a24acede1c7983e8a2d2edd2f0b024bbcf5c01e5f093e49f"}
Oct 30 04:36:09.182782 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:09.182675    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89" event={"ID":"7dabf1f5-a781-45e8-a9f2-622109ab647b","Type":"ContainerStarted","Data":"e06b9ae26e61c517f3fcb06971df6208afe882a068f8e7e50e67782c5a78617b"}
Oct 30 04:36:09.227426 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:09.227362    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89" podStartSLOduration=1.22734407 podStartE2EDuration="1.22734407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:36:09.225779591 +0000 UTC m=+2288.638175187" watchObservedRunningTime="2026-10-30 04:36:09.22734407 +0000 UTC m=+2288.639739646"
Oct 30 04:36:09.660808 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:09.660764    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:10.662669 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:10.662637    4874 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 30 04:36:10.663047 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:10.662686    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89" podUID="7dabf1f5-a781-45e8-a9f2-622109ab647b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 30 04:36:10.852717 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:10.852684    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"]
Oct 30 04:36:10.852911 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:10.852881    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4" podUID="bc017d56-de5c-4ce1-bb57-421f757c790e" containerName="istio-proxy" containerID="cri-o://90c85095f32956eb5d92da84fa92178e1075044114cdf456d75747075bed6547" gracePeriod=30
Oct 30 04:36:11.660784 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:11.660754    4874 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 30 04:36:11.660960 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:11.660811    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89" podUID="7dabf1f5-a781-45e8-a9f2-622109ab647b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 30 04:36:12.661554 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:12.661517    4874 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 30 04:36:12.661933 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:12.661594    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89" podUID="7dabf1f5-a781-45e8-a9f2-622109ab647b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 30 04:36:13.660758 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:13.660680    4874 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 30 04:36:13.660758 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:13.660731    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89" podUID="7dabf1f5-a781-45e8-a9f2-622109ab647b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 30 04:36:14.661048 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:14.661013    4874 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 30 04:36:14.661463 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:14.661064    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89" podUID="7dabf1f5-a781-45e8-a9f2-622109ab647b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 30 04:36:15.847937 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:15.847902    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:15.849264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:15.848688    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:15.849657 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:15.849633    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:36:15.955724 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:15.955692    4874 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 04:36:15.955851 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:15.955738    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4" podUID="bc017d56-de5c-4ce1-bb57-421f757c790e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 04:36:16.130839 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.130817    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:36:16.227334 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.227302    4874 generic.go:356] "Generic (PLEG): container finished" podID="bc017d56-de5c-4ce1-bb57-421f757c790e" containerID="90c85095f32956eb5d92da84fa92178e1075044114cdf456d75747075bed6547" exitCode=0
Oct 30 04:36:16.227522 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.227354    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"
Oct 30 04:36:16.227522 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.227397    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4" event={"ID":"bc017d56-de5c-4ce1-bb57-421f757c790e","Type":"ContainerDied","Data":"90c85095f32956eb5d92da84fa92178e1075044114cdf456d75747075bed6547"}
Oct 30 04:36:16.227522 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.227444    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4" event={"ID":"bc017d56-de5c-4ce1-bb57-421f757c790e","Type":"ContainerDied","Data":"4379e0b3d15dceb169af4249e2c26510109bfe336cdbc7eecd3e50bf4f958b61"}
Oct 30 04:36:16.227522 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.227467    4874 scope.go:122] "RemoveContainer" containerID="90c85095f32956eb5d92da84fa92178e1075044114cdf456d75747075bed6547"
Oct 30 04:36:16.236744 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.236725    4874 scope.go:122] "RemoveContainer" containerID="90c85095f32956eb5d92da84fa92178e1075044114cdf456d75747075bed6547"
Oct 30 04:36:16.237031 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:36:16.237009    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90c85095f32956eb5d92da84fa92178e1075044114cdf456d75747075bed6547\": container with ID starting with 90c85095f32956eb5d92da84fa92178e1075044114cdf456d75747075bed6547 not found: ID does not exist" containerID="90c85095f32956eb5d92da84fa92178e1075044114cdf456d75747075bed6547"
Oct 30 04:36:16.237124 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.237041    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c85095f32956eb5d92da84fa92178e1075044114cdf456d75747075bed6547"} err="failed to get container status \"90c85095f32956eb5d92da84fa92178e1075044114cdf456d75747075bed6547\": rpc error: code = NotFound desc = could not find container \"90c85095f32956eb5d92da84fa92178e1075044114cdf456d75747075bed6547\": container with ID starting with 90c85095f32956eb5d92da84fa92178e1075044114cdf456d75747075bed6547 not found: ID does not exist"
Oct 30 04:36:16.250050 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.250030    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-credential-socket\") pod \"bc017d56-de5c-4ce1-bb57-421f757c790e\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") "
Oct 30 04:36:16.250133 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.250059    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-workload-socket\") pod \"bc017d56-de5c-4ce1-bb57-421f757c790e\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") "
Oct 30 04:36:16.250133 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.250094    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-data\") pod \"bc017d56-de5c-4ce1-bb57-421f757c790e\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") "
Oct 30 04:36:16.250133 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.250118    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-envoy\") pod \"bc017d56-de5c-4ce1-bb57-421f757c790e\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") "
Oct 30 04:36:16.250336 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.250138    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-workload-certs\") pod \"bc017d56-de5c-4ce1-bb57-421f757c790e\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") "
Oct 30 04:36:16.250336 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.250198    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-podinfo\") pod \"bc017d56-de5c-4ce1-bb57-421f757c790e\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") "
Oct 30 04:36:16.250336 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.250326    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc017d56-de5c-4ce1-bb57-421f757c790e-kube-api-access-62n4x\" (UniqueName: \"kubernetes.io/projected/bc017d56-de5c-4ce1-bb57-421f757c790e-kube-api-access-62n4x\") pod \"bc017d56-de5c-4ce1-bb57-421f757c790e\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") "
Oct 30 04:36:16.250488 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.250361    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc017d56-de5c-4ce1-bb57-421f757c790e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc017d56-de5c-4ce1-bb57-421f757c790e-istiod-ca-cert\") pod \"bc017d56-de5c-4ce1-bb57-421f757c790e\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") "
Oct 30 04:36:16.250488 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.250347    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-credential-socket" pod "bc017d56-de5c-4ce1-bb57-421f757c790e" (UID: "bc017d56-de5c-4ce1-bb57-421f757c790e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:36:16.250488 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.250409    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-workload-socket" pod "bc017d56-de5c-4ce1-bb57-421f757c790e" (UID: "bc017d56-de5c-4ce1-bb57-421f757c790e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:36:16.250488 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.250420    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-data" pod "bc017d56-de5c-4ce1-bb57-421f757c790e" (UID: "bc017d56-de5c-4ce1-bb57-421f757c790e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:36:16.250488 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.250482    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-token\" (UniqueName: \"kubernetes.io/projected/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-token\") pod \"bc017d56-de5c-4ce1-bb57-421f757c790e\" (UID: \"bc017d56-de5c-4ce1-bb57-421f757c790e\") "
Oct 30 04:36:16.250722 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.250656    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-workload-certs" pod "bc017d56-de5c-4ce1-bb57-421f757c790e" (UID: "bc017d56-de5c-4ce1-bb57-421f757c790e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:36:16.250772 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.250756    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc017d56-de5c-4ce1-bb57-421f757c790e-istiod-ca-cert" pod "bc017d56-de5c-4ce1-bb57-421f757c790e" (UID: "bc017d56-de5c-4ce1-bb57-421f757c790e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:36:16.251041 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.251018    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:36:16.251126 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.251045    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:36:16.251126 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.251059    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:36:16.251126 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.251072    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:36:16.251126 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.251085    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc017d56-de5c-4ce1-bb57-421f757c790e-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:36:16.252564 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.252536    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-envoy" pod "bc017d56-de5c-4ce1-bb57-421f757c790e" (UID: "bc017d56-de5c-4ce1-bb57-421f757c790e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:36:16.252650 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.252616    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-token" pod "bc017d56-de5c-4ce1-bb57-421f757c790e" (UID: "bc017d56-de5c-4ce1-bb57-421f757c790e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:36:16.252876 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.252854    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-podinfo" pod "bc017d56-de5c-4ce1-bb57-421f757c790e" (UID: "bc017d56-de5c-4ce1-bb57-421f757c790e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:36:16.252931 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.252882    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc017d56-de5c-4ce1-bb57-421f757c790e-kube-api-access-62n4x" pod "bc017d56-de5c-4ce1-bb57-421f757c790e" (UID: "bc017d56-de5c-4ce1-bb57-421f757c790e"). InnerVolumeSpecName "kube-api-access-62n4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:36:16.352343 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.352308    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62n4x\" (UniqueName: \"kubernetes.io/projected/bc017d56-de5c-4ce1-bb57-421f757c790e-kube-api-access-62n4x\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:36:16.352343 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.352338    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:36:16.352546 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.352355    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:36:16.352546 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.352368    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc017d56-de5c-4ce1-bb57-421f757c790e-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:36:16.577942 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.577909    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"]
Oct 30 04:36:16.585002 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.584977    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-svdg4"]
Oct 30 04:36:16.729537 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:36:16.729499    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc017d56-de5c-4ce1-bb57-421f757c790e" path="/var/lib/kubelet/pods/bc017d56-de5c-4ce1-bb57-421f757c790e/volumes"
Oct 30 04:36:25.608908 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:36:25.608871    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:36:55.608624 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:36:55.608588    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:37:25.609510 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:37:25.609472    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:37:55.608715 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:37:55.608675    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:38:00.956493 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:00.956467    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 04:38:00.990259 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:00.990217    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 04:38:25.608448 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:38:25.608410    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:38:41.289946 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.289912    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"]
Oct 30 04:38:41.290380 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.290366    4874 cpu_manager.go:414] "containerMap: removing container" podUID="bc017d56-de5c-4ce1-bb57-421f757c790e" containerName="istio-proxy"
Oct 30 04:38:41.290380 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.290378    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc017d56-de5c-4ce1-bb57-421f757c790e" containerName="istio-proxy"
Oct 30 04:38:41.290509 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.290498    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc017d56-de5c-4ce1-bb57-421f757c790e" containerName="istio-proxy"
Oct 30 04:38:41.293586 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.293557    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.308215 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.308191    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-6796b1-openshift-default-dockercfg-h254p\""
Oct 30 04:38:41.324648 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.324621    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"]
Oct 30 04:38:41.375607 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.375576    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.375607 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.375606    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.375788 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.375634    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.375788 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.375754    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.375859 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.375793    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.375892 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.375852    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.375994 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.375980    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmkcq\" (UniqueName: \"kubernetes.io/projected/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-kube-api-access-lmkcq\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.376030 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.376001    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.376030 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.376025    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.477048 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.477020    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.477166 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.477055    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.477166 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.477077    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.477166 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.477098    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.477354 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.477167    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmkcq\" (UniqueName: \"kubernetes.io/projected/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-kube-api-access-lmkcq\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.477354 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.477187    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.477354 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.477331    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.477518 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.477416    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.477518 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.477444    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.477518 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.477495    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.477672 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.477554    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.477760 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.477736    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.477822 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.477761    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.478086 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.478070    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.479445 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.479419    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.479969 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.479948    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.501371 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.501350    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.501601 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.501582    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmkcq\" (UniqueName: \"kubernetes.io/projected/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-kube-api-access-lmkcq\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.607444 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.607388    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:41.928434 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.928408    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"]
Oct 30 04:38:41.931382 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:38:41.931355    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee079d02_8eaf_4b69_9f04_ffdb2f5073a2.slice/crio-d40f22c63fc4020c549d877025ae26c34a3937c2fa41d4be1ce7d7d8bd0e1dd7 WatchSource:0}: Error finding container d40f22c63fc4020c549d877025ae26c34a3937c2fa41d4be1ce7d7d8bd0e1dd7: Status 404 returned error can't find the container with id d40f22c63fc4020c549d877025ae26c34a3937c2fa41d4be1ce7d7d8bd0e1dd7
Oct 30 04:38:41.933539 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.933506    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:38:41.933609 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.933574    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:38:41.933646 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.933604    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:38:41.986156 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:41.986125    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz" event={"ID":"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2","Type":"ContainerStarted","Data":"d40f22c63fc4020c549d877025ae26c34a3937c2fa41d4be1ce7d7d8bd0e1dd7"}
Oct 30 04:38:42.996052 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:42.996010    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz" event={"ID":"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2","Type":"ContainerStarted","Data":"c8f8d9f2a6b1b86b0beb50dbf66d75de521ece8dfeba5dbda7f6a1e8cde905aa"}
Oct 30 04:38:43.072554 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:43.072477    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz" podStartSLOduration=2.072456927 podStartE2EDuration="2.072456927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:38:43.058357947 +0000 UTC m=+2442.470753522" watchObservedRunningTime="2026-10-30 04:38:43.072456927 +0000 UTC m=+2442.484852502"
Oct 30 04:38:43.607770 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:43.607734    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:43.609418 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:43.609393    4874 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 30 04:38:43.609538 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:43.609442    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz" podUID="ee079d02-8eaf-4b69-9f04-ffdb2f5073a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 30 04:38:44.608815 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:44.608777    4874 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 30 04:38:44.609217 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:44.608834    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz" podUID="ee079d02-8eaf-4b69-9f04-ffdb2f5073a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 30 04:38:45.608311 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:45.608277    4874 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 30 04:38:45.608544 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:45.608519    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz" podUID="ee079d02-8eaf-4b69-9f04-ffdb2f5073a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 30 04:38:46.607895 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:46.607860    4874 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 30 04:38:46.608351 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:46.608320    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz" podUID="ee079d02-8eaf-4b69-9f04-ffdb2f5073a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 30 04:38:47.607922 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:47.607887    4874 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 30 04:38:47.608386 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:47.608352    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz" podUID="ee079d02-8eaf-4b69-9f04-ffdb2f5073a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 30 04:38:48.775763 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:48.775724    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:48.776659 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:48.776625    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:48.777461 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:48.777440    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:38:54.342463 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:54.342424    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"]
Oct 30 04:38:54.342924 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:54.342661    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28" podUID="4ccf3fdf-09b2-48b2-a270-2500fe34b892" containerName="istio-proxy" containerID="cri-o://47c918762888c2cd75a3a699e31a44d2e520c34dc1663aea426558abd487af03" gracePeriod=30
Oct 30 04:38:55.608293 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:38:55.608259    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:38:59.601197 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.601171    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:38:59.686580 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.686506    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-podinfo\") pod \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") "
Oct 30 04:38:59.686580 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.686556    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-envoy\") pod \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") "
Oct 30 04:38:59.686803 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.686668    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-credential-socket\") pod \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") "
Oct 30 04:38:59.686803 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.686728    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istiod-ca-cert\") pod \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") "
Oct 30 04:38:59.686803 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.686797    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-workload-socket\") pod \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") "
Oct 30 04:38:59.686963 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.686831    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-workload-certs\") pod \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") "
Oct 30 04:38:59.686963 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.686864    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-token\" (UniqueName: \"kubernetes.io/projected/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-token\") pod \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") "
Oct 30 04:38:59.686963 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.686894    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ccf3fdf-09b2-48b2-a270-2500fe34b892-kube-api-access-8z9w6\" (UniqueName: \"kubernetes.io/projected/4ccf3fdf-09b2-48b2-a270-2500fe34b892-kube-api-access-8z9w6\") pod \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") "
Oct 30 04:38:59.686963 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.686953    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-data\") pod \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\" (UID: \"4ccf3fdf-09b2-48b2-a270-2500fe34b892\") "
Oct 30 04:38:59.687180 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.686961    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-credential-socket" pod "4ccf3fdf-09b2-48b2-a270-2500fe34b892" (UID: "4ccf3fdf-09b2-48b2-a270-2500fe34b892"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:38:59.687180 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.687113    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-workload-certs" pod "4ccf3fdf-09b2-48b2-a270-2500fe34b892" (UID: "4ccf3fdf-09b2-48b2-a270-2500fe34b892"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:38:59.687335 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.687198    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-workload-socket" pod "4ccf3fdf-09b2-48b2-a270-2500fe34b892" (UID: "4ccf3fdf-09b2-48b2-a270-2500fe34b892"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:38:59.687335 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.687133    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istiod-ca-cert" pod "4ccf3fdf-09b2-48b2-a270-2500fe34b892" (UID: "4ccf3fdf-09b2-48b2-a270-2500fe34b892"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:38:59.687483 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.687461    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-data" pod "4ccf3fdf-09b2-48b2-a270-2500fe34b892" (UID: "4ccf3fdf-09b2-48b2-a270-2500fe34b892"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:38:59.687725 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.687634    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:38:59.687725 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.687660    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:38:59.687725 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.687674    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:38:59.687725 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.687688    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:38:59.687725 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.687702    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:38:59.689397 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.689361    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-envoy" pod "4ccf3fdf-09b2-48b2-a270-2500fe34b892" (UID: "4ccf3fdf-09b2-48b2-a270-2500fe34b892"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:38:59.690622 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.689771    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ccf3fdf-09b2-48b2-a270-2500fe34b892-kube-api-access-8z9w6" pod "4ccf3fdf-09b2-48b2-a270-2500fe34b892" (UID: "4ccf3fdf-09b2-48b2-a270-2500fe34b892"). InnerVolumeSpecName "kube-api-access-8z9w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:38:59.690966 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.690928    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-podinfo" pod "4ccf3fdf-09b2-48b2-a270-2500fe34b892" (UID: "4ccf3fdf-09b2-48b2-a270-2500fe34b892"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:38:59.691079 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.691055    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-token" pod "4ccf3fdf-09b2-48b2-a270-2500fe34b892" (UID: "4ccf3fdf-09b2-48b2-a270-2500fe34b892"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:38:59.788994 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.788965    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:38:59.788994 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.788991    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z9w6\" (UniqueName: \"kubernetes.io/projected/4ccf3fdf-09b2-48b2-a270-2500fe34b892-kube-api-access-8z9w6\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:38:59.789150 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.789011    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:38:59.789150 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:38:59.789023    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ccf3fdf-09b2-48b2-a270-2500fe34b892-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:39:00.083958 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:00.083924    4874 generic.go:356] "Generic (PLEG): container finished" podID="4ccf3fdf-09b2-48b2-a270-2500fe34b892" containerID="47c918762888c2cd75a3a699e31a44d2e520c34dc1663aea426558abd487af03" exitCode=0
Oct 30 04:39:00.084125 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:00.083978    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"
Oct 30 04:39:00.084125 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:00.084011    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28" event={"ID":"4ccf3fdf-09b2-48b2-a270-2500fe34b892","Type":"ContainerDied","Data":"47c918762888c2cd75a3a699e31a44d2e520c34dc1663aea426558abd487af03"}
Oct 30 04:39:00.084125 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:00.084100    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28" event={"ID":"4ccf3fdf-09b2-48b2-a270-2500fe34b892","Type":"ContainerDied","Data":"8fccff1f6d20d080bb8c69f93343282cebf55836f874d493f6d32a3a2cf88ee8"}
Oct 30 04:39:00.084125 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:00.084123    4874 scope.go:122] "RemoveContainer" containerID="47c918762888c2cd75a3a699e31a44d2e520c34dc1663aea426558abd487af03"
Oct 30 04:39:00.093485 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:00.093465    4874 scope.go:122] "RemoveContainer" containerID="47c918762888c2cd75a3a699e31a44d2e520c34dc1663aea426558abd487af03"
Oct 30 04:39:00.093741 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:39:00.093722    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c918762888c2cd75a3a699e31a44d2e520c34dc1663aea426558abd487af03\": container with ID starting with 47c918762888c2cd75a3a699e31a44d2e520c34dc1663aea426558abd487af03 not found: ID does not exist" containerID="47c918762888c2cd75a3a699e31a44d2e520c34dc1663aea426558abd487af03"
Oct 30 04:39:00.093802 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:00.093754    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c918762888c2cd75a3a699e31a44d2e520c34dc1663aea426558abd487af03"} err="failed to get container status \"47c918762888c2cd75a3a699e31a44d2e520c34dc1663aea426558abd487af03\": rpc error: code = NotFound desc = could not find container \"47c918762888c2cd75a3a699e31a44d2e520c34dc1663aea426558abd487af03\": container with ID starting with 47c918762888c2cd75a3a699e31a44d2e520c34dc1663aea426558abd487af03 not found: ID does not exist"
Oct 30 04:39:00.133206 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:00.133183    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"]
Oct 30 04:39:00.139087 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:00.139065    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-d4g28"]
Oct 30 04:39:00.729620 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:00.729589    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ccf3fdf-09b2-48b2-a270-2500fe34b892" path="/var/lib/kubelet/pods/4ccf3fdf-09b2-48b2-a270-2500fe34b892/volumes"
Oct 30 04:39:09.383032 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:09.382993    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"]
Oct 30 04:39:09.383495 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:09.383186    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2" podUID="eaf6c9aa-0b1b-42ef-acca-f434e0e526c1" containerName="istio-proxy" containerID="cri-o://bcd957a378f93c5ac76259854c636380d20bb6b9f8a73a4cde595dad6bef493c" gracePeriod=30
Oct 30 04:39:14.636788 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.636764    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:39:14.648894 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.648839    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-kube-api-access-l6cmw\" (UniqueName: \"kubernetes.io/projected/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-kube-api-access-l6cmw\") pod \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") "
Oct 30 04:39:14.648980 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.648894    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-envoy\") pod \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") "
Oct 30 04:39:14.649100 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.649076    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-workload-certs\") pod \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") "
Oct 30 04:39:14.649182 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.649162    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-token\" (UniqueName: \"kubernetes.io/projected/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-token\") pod \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") "
Oct 30 04:39:14.649308 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.649286    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istiod-ca-cert\") pod \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") "
Oct 30 04:39:14.649397 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.649340    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-podinfo\") pod \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") "
Oct 30 04:39:14.649456 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.649372    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-workload-certs" pod "eaf6c9aa-0b1b-42ef-acca-f434e0e526c1" (UID: "eaf6c9aa-0b1b-42ef-acca-f434e0e526c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:39:14.649456 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.649396    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-data\") pod \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") "
Oct 30 04:39:14.649559 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.649545    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-credential-socket\") pod \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") "
Oct 30 04:39:14.649615 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.649590    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-data" pod "eaf6c9aa-0b1b-42ef-acca-f434e0e526c1" (UID: "eaf6c9aa-0b1b-42ef-acca-f434e0e526c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:39:14.649615 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.649607    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-workload-socket\") pod \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\" (UID: \"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1\") "
Oct 30 04:39:14.649808 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.649789    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-credential-socket" pod "eaf6c9aa-0b1b-42ef-acca-f434e0e526c1" (UID: "eaf6c9aa-0b1b-42ef-acca-f434e0e526c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:39:14.649874 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.649719    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istiod-ca-cert" pod "eaf6c9aa-0b1b-42ef-acca-f434e0e526c1" (UID: "eaf6c9aa-0b1b-42ef-acca-f434e0e526c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:39:14.650015 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.649996    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-workload-socket" pod "eaf6c9aa-0b1b-42ef-acca-f434e0e526c1" (UID: "eaf6c9aa-0b1b-42ef-acca-f434e0e526c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:39:14.650078 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.650058    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:39:14.650078 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.650072    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:39:14.650155 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.650081    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:39:14.650155 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.650089    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:39:14.651452 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.651431    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-envoy" pod "eaf6c9aa-0b1b-42ef-acca-f434e0e526c1" (UID: "eaf6c9aa-0b1b-42ef-acca-f434e0e526c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:39:14.651528 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.651447    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-kube-api-access-l6cmw" pod "eaf6c9aa-0b1b-42ef-acca-f434e0e526c1" (UID: "eaf6c9aa-0b1b-42ef-acca-f434e0e526c1"). InnerVolumeSpecName "kube-api-access-l6cmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:39:14.651671 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.651648    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-token" pod "eaf6c9aa-0b1b-42ef-acca-f434e0e526c1" (UID: "eaf6c9aa-0b1b-42ef-acca-f434e0e526c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:39:14.651782 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.651761    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-podinfo" pod "eaf6c9aa-0b1b-42ef-acca-f434e0e526c1" (UID: "eaf6c9aa-0b1b-42ef-acca-f434e0e526c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:39:14.750760 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.750732    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:39:14.750760 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.750759    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6cmw\" (UniqueName: \"kubernetes.io/projected/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-kube-api-access-l6cmw\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:39:14.750927 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.750770    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:39:14.750927 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.750778    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:39:14.750927 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:14.750786    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:39:15.169805 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:15.169772    4874 generic.go:356] "Generic (PLEG): container finished" podID="eaf6c9aa-0b1b-42ef-acca-f434e0e526c1" containerID="bcd957a378f93c5ac76259854c636380d20bb6b9f8a73a4cde595dad6bef493c" exitCode=0
Oct 30 04:39:15.170000 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:15.169821    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"
Oct 30 04:39:15.170000 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:15.169845    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2" event={"ID":"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1","Type":"ContainerDied","Data":"bcd957a378f93c5ac76259854c636380d20bb6b9f8a73a4cde595dad6bef493c"}
Oct 30 04:39:15.170000 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:15.169887    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2" event={"ID":"eaf6c9aa-0b1b-42ef-acca-f434e0e526c1","Type":"ContainerDied","Data":"5ef2da3bd86e3b45fde79098895a13fc527f1e78e135bc7e151cef402ec4d9db"}
Oct 30 04:39:15.170000 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:15.169909    4874 scope.go:122] "RemoveContainer" containerID="bcd957a378f93c5ac76259854c636380d20bb6b9f8a73a4cde595dad6bef493c"
Oct 30 04:39:15.178928 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:15.178907    4874 scope.go:122] "RemoveContainer" containerID="bcd957a378f93c5ac76259854c636380d20bb6b9f8a73a4cde595dad6bef493c"
Oct 30 04:39:15.179202 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:39:15.179182    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcd957a378f93c5ac76259854c636380d20bb6b9f8a73a4cde595dad6bef493c\": container with ID starting with bcd957a378f93c5ac76259854c636380d20bb6b9f8a73a4cde595dad6bef493c not found: ID does not exist" containerID="bcd957a378f93c5ac76259854c636380d20bb6b9f8a73a4cde595dad6bef493c"
Oct 30 04:39:15.179268 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:15.179210    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcd957a378f93c5ac76259854c636380d20bb6b9f8a73a4cde595dad6bef493c"} err="failed to get container status \"bcd957a378f93c5ac76259854c636380d20bb6b9f8a73a4cde595dad6bef493c\": rpc error: code = NotFound desc = could not find container \"bcd957a378f93c5ac76259854c636380d20bb6b9f8a73a4cde595dad6bef493c\": container with ID starting with bcd957a378f93c5ac76259854c636380d20bb6b9f8a73a4cde595dad6bef493c not found: ID does not exist"
Oct 30 04:39:15.230670 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:15.230648    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"]
Oct 30 04:39:15.241974 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:15.241954    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-447h2"]
Oct 30 04:39:15.739642 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:15.739611    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"]
Oct 30 04:39:15.740022 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:15.739786    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt" podUID="3d59c727-0c93-401b-8f3f-75a568a3bb32" containerName="istio-proxy" containerID="cri-o://41031e7e413851942c4d079e097b48397833128987a10690a87752dee8fcdb6f" gracePeriod=30
Oct 30 04:39:16.729156 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:16.729122    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaf6c9aa-0b1b-42ef-acca-f434e0e526c1" path="/var/lib/kubelet/pods/eaf6c9aa-0b1b-42ef-acca-f434e0e526c1/volumes"
Oct 30 04:39:20.995338 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:20.995315    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:39:21.012784 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.012728    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d59c727-0c93-401b-8f3f-75a568a3bb32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d59c727-0c93-401b-8f3f-75a568a3bb32-istiod-ca-cert\") pod \"3d59c727-0c93-401b-8f3f-75a568a3bb32\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") "
Oct 30 04:39:21.012896 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.012781    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-credential-socket\") pod \"3d59c727-0c93-401b-8f3f-75a568a3bb32\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") "
Oct 30 04:39:21.012896 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.012805    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d59c727-0c93-401b-8f3f-75a568a3bb32-kube-api-access-sqkc4\" (UniqueName: \"kubernetes.io/projected/3d59c727-0c93-401b-8f3f-75a568a3bb32-kube-api-access-sqkc4\") pod \"3d59c727-0c93-401b-8f3f-75a568a3bb32\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") "
Oct 30 04:39:21.012896 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.012864    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-envoy\") pod \"3d59c727-0c93-401b-8f3f-75a568a3bb32\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") "
Oct 30 04:39:21.013066 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.012918    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-data\") pod \"3d59c727-0c93-401b-8f3f-75a568a3bb32\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") "
Oct 30 04:39:21.013066 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.012961    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-podinfo\") pod \"3d59c727-0c93-401b-8f3f-75a568a3bb32\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") "
Oct 30 04:39:21.013066 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.013034    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-token\" (UniqueName: \"kubernetes.io/projected/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-token\") pod \"3d59c727-0c93-401b-8f3f-75a568a3bb32\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") "
Oct 30 04:39:21.013229 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.013091    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-workload-certs\") pod \"3d59c727-0c93-401b-8f3f-75a568a3bb32\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") "
Oct 30 04:39:21.013229 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.013101    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d59c727-0c93-401b-8f3f-75a568a3bb32-istiod-ca-cert" pod "3d59c727-0c93-401b-8f3f-75a568a3bb32" (UID: "3d59c727-0c93-401b-8f3f-75a568a3bb32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:39:21.013229 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.013117    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-workload-socket\") pod \"3d59c727-0c93-401b-8f3f-75a568a3bb32\" (UID: \"3d59c727-0c93-401b-8f3f-75a568a3bb32\") "
Oct 30 04:39:21.013229 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.013120    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-credential-socket" pod "3d59c727-0c93-401b-8f3f-75a568a3bb32" (UID: "3d59c727-0c93-401b-8f3f-75a568a3bb32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:39:21.013459 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.013226    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-data" pod "3d59c727-0c93-401b-8f3f-75a568a3bb32" (UID: "3d59c727-0c93-401b-8f3f-75a568a3bb32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:39:21.013459 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.013426    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-workload-socket" pod "3d59c727-0c93-401b-8f3f-75a568a3bb32" (UID: "3d59c727-0c93-401b-8f3f-75a568a3bb32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:39:21.013610 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.013587    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-workload-certs" pod "3d59c727-0c93-401b-8f3f-75a568a3bb32" (UID: "3d59c727-0c93-401b-8f3f-75a568a3bb32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:39:21.013672 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.013609    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:39:21.013672 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.013629    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d59c727-0c93-401b-8f3f-75a568a3bb32-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:39:21.013672 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.013643    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:39:21.013672 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.013656    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:39:21.015300 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.015270    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-token" pod "3d59c727-0c93-401b-8f3f-75a568a3bb32" (UID: "3d59c727-0c93-401b-8f3f-75a568a3bb32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:39:21.015403 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.015332    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-podinfo" pod "3d59c727-0c93-401b-8f3f-75a568a3bb32" (UID: "3d59c727-0c93-401b-8f3f-75a568a3bb32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:39:21.015517 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.015500    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-envoy" pod "3d59c727-0c93-401b-8f3f-75a568a3bb32" (UID: "3d59c727-0c93-401b-8f3f-75a568a3bb32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:39:21.015678 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.015658    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d59c727-0c93-401b-8f3f-75a568a3bb32-kube-api-access-sqkc4" pod "3d59c727-0c93-401b-8f3f-75a568a3bb32" (UID: "3d59c727-0c93-401b-8f3f-75a568a3bb32"). InnerVolumeSpecName "kube-api-access-sqkc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:39:21.114519 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.114489    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:39:21.114519 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.114515    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:39:21.114519 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.114524    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d59c727-0c93-401b-8f3f-75a568a3bb32-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:39:21.114719 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.114532    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d59c727-0c93-401b-8f3f-75a568a3bb32-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:39:21.114719 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.114542    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqkc4\" (UniqueName: \"kubernetes.io/projected/3d59c727-0c93-401b-8f3f-75a568a3bb32-kube-api-access-sqkc4\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:39:21.202681 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.202647    4874 generic.go:356] "Generic (PLEG): container finished" podID="3d59c727-0c93-401b-8f3f-75a568a3bb32" containerID="41031e7e413851942c4d079e097b48397833128987a10690a87752dee8fcdb6f" exitCode=0
Oct 30 04:39:21.202842 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.202697    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"
Oct 30 04:39:21.202842 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.202722    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt" event={"ID":"3d59c727-0c93-401b-8f3f-75a568a3bb32","Type":"ContainerDied","Data":"41031e7e413851942c4d079e097b48397833128987a10690a87752dee8fcdb6f"}
Oct 30 04:39:21.202842 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.202775    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt" event={"ID":"3d59c727-0c93-401b-8f3f-75a568a3bb32","Type":"ContainerDied","Data":"63e766e75f8770c85595f5964ae5606edb00ef27d90bf9fa00551afbbee6adcd"}
Oct 30 04:39:21.202842 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.202796    4874 scope.go:122] "RemoveContainer" containerID="41031e7e413851942c4d079e097b48397833128987a10690a87752dee8fcdb6f"
Oct 30 04:39:21.212458 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.212438    4874 scope.go:122] "RemoveContainer" containerID="41031e7e413851942c4d079e097b48397833128987a10690a87752dee8fcdb6f"
Oct 30 04:39:21.212718 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:39:21.212702    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41031e7e413851942c4d079e097b48397833128987a10690a87752dee8fcdb6f\": container with ID starting with 41031e7e413851942c4d079e097b48397833128987a10690a87752dee8fcdb6f not found: ID does not exist" containerID="41031e7e413851942c4d079e097b48397833128987a10690a87752dee8fcdb6f"
Oct 30 04:39:21.212788 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.212730    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41031e7e413851942c4d079e097b48397833128987a10690a87752dee8fcdb6f"} err="failed to get container status \"41031e7e413851942c4d079e097b48397833128987a10690a87752dee8fcdb6f\": rpc error: code = NotFound desc = could not find container \"41031e7e413851942c4d079e097b48397833128987a10690a87752dee8fcdb6f\": container with ID starting with 41031e7e413851942c4d079e097b48397833128987a10690a87752dee8fcdb6f not found: ID does not exist"
Oct 30 04:39:21.252341 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.252318    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"]
Oct 30 04:39:21.256679 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:21.256657    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-skkgt"]
Oct 30 04:39:22.728449 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:22.728415    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d59c727-0c93-401b-8f3f-75a568a3bb32" path="/var/lib/kubelet/pods/3d59c727-0c93-401b-8f3f-75a568a3bb32/volumes"
Oct 30 04:39:25.560486 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:25.560448    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:39:25.676686 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:39:25.671849    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:39:30.206020 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.205990    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"]
Oct 30 04:39:30.206559 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.206448    4874 cpu_manager.go:414] "containerMap: removing container" podUID="3d59c727-0c93-401b-8f3f-75a568a3bb32" containerName="istio-proxy"
Oct 30 04:39:30.206559 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.206462    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d59c727-0c93-401b-8f3f-75a568a3bb32" containerName="istio-proxy"
Oct 30 04:39:30.206559 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.206485    4874 cpu_manager.go:414] "containerMap: removing container" podUID="eaf6c9aa-0b1b-42ef-acca-f434e0e526c1" containerName="istio-proxy"
Oct 30 04:39:30.206559 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.206490    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf6c9aa-0b1b-42ef-acca-f434e0e526c1" containerName="istio-proxy"
Oct 30 04:39:30.206559 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.206525    4874 cpu_manager.go:414] "containerMap: removing container" podUID="4ccf3fdf-09b2-48b2-a270-2500fe34b892" containerName="istio-proxy"
Oct 30 04:39:30.206559 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.206531    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ccf3fdf-09b2-48b2-a270-2500fe34b892" containerName="istio-proxy"
Oct 30 04:39:30.206897 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.206594    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ccf3fdf-09b2-48b2-a270-2500fe34b892" containerName="istio-proxy"
Oct 30 04:39:30.206897 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.206601    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d59c727-0c93-401b-8f3f-75a568a3bb32" containerName="istio-proxy"
Oct 30 04:39:30.206897 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.206610    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf6c9aa-0b1b-42ef-acca-f434e0e526c1" containerName="istio-proxy"
Oct 30 04:39:30.209877 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.209845    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.225692 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.225664    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-e070ea-openshift-default-dockercfg-9bq8n\""
Oct 30 04:39:30.241773 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.241748    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"]
Oct 30 04:39:30.332101 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.331064    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-workload-certs\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.332101 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.331172    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-istio-envoy\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.332101 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.331228    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9cgw\" (UniqueName: \"kubernetes.io/projected/833c472c-0226-4e47-bc11-5d2659845a86-kube-api-access-j9cgw\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.332101 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.331280    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/833c472c-0226-4e47-bc11-5d2659845a86-istio-token\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.332101 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.331308    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-workload-socket\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.332101 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.331364    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/833c472c-0226-4e47-bc11-5d2659845a86-istio-podinfo\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.332101 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.331478    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/833c472c-0226-4e47-bc11-5d2659845a86-istiod-ca-cert\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.332101 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.331562    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-credential-socket\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.332101 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.331600    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-istio-data\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.433192 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.433150    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-credential-socket\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.433192 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.433192    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-istio-data\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.433469 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.433229    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-workload-certs\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.433469 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.433258    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-istio-envoy\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.433469 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.433293    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9cgw\" (UniqueName: \"kubernetes.io/projected/833c472c-0226-4e47-bc11-5d2659845a86-kube-api-access-j9cgw\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.433469 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.433323    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/833c472c-0226-4e47-bc11-5d2659845a86-istio-token\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.433469 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.433350    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-workload-socket\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.433469 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.433400    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/833c472c-0226-4e47-bc11-5d2659845a86-istio-podinfo\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.433798 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.433497    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/833c472c-0226-4e47-bc11-5d2659845a86-istiod-ca-cert\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.433798 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.433612    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-credential-socket\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.433798 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.433695    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-istio-data\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.433963 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.433907    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-workload-socket\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.434207 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.434185    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/833c472c-0226-4e47-bc11-5d2659845a86-istiod-ca-cert\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.434311 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.434219    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-workload-certs\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.435710 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.435692    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-istio-envoy\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.435917 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.435899    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/833c472c-0226-4e47-bc11-5d2659845a86-istio-podinfo\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.455699 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.455677    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/833c472c-0226-4e47-bc11-5d2659845a86-istio-token\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.455927 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.455910    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9cgw\" (UniqueName: \"kubernetes.io/projected/833c472c-0226-4e47-bc11-5d2659845a86-kube-api-access-j9cgw\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.524026 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.524001    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:30.878098 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.877904    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"]
Oct 30 04:39:30.880928 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:39:30.880893    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833c472c_0226_4e47_bc11_5d2659845a86.slice/crio-cd2f2e6203a64b3056232925dbfb780bbc43535e907e169d361e86666d9f9fd5 WatchSource:0}: Error finding container cd2f2e6203a64b3056232925dbfb780bbc43535e907e169d361e86666d9f9fd5: Status 404 returned error can't find the container with id cd2f2e6203a64b3056232925dbfb780bbc43535e907e169d361e86666d9f9fd5
Oct 30 04:39:30.883195 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.883162    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:39:30.883361 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.883251    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:39:30.883361 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:30.883317    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:39:31.254733 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:31.254701    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs" event={"ID":"833c472c-0226-4e47-bc11-5d2659845a86","Type":"ContainerStarted","Data":"ca61199c97394dfa8d2e03da7988c0080ccbbee9721005442a3ca2dbe62eb7c8"}
Oct 30 04:39:31.255106 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:31.254751    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs" event={"ID":"833c472c-0226-4e47-bc11-5d2659845a86","Type":"ContainerStarted","Data":"cd2f2e6203a64b3056232925dbfb780bbc43535e907e169d361e86666d9f9fd5"}
Oct 30 04:39:31.323875 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:31.323829    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs" podStartSLOduration=1.323815782 podStartE2EDuration="1.323815782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:39:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:39:31.311130423 +0000 UTC m=+2490.723525996" watchObservedRunningTime="2026-10-30 04:39:31.323815782 +0000 UTC m=+2490.736211355"
Oct 30 04:39:31.524634 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:31.524550    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:31.530001 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:31.529978    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:32.274550 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:32.274511    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:32.275518 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:32.275498    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:39:34.185668 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.185630    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"]
Oct 30 04:39:34.189944 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.189921    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.207191 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.207170    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-66217b-openshift-default-dockercfg-vhs9s\""
Oct 30 04:39:34.220290 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.220264    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"]
Oct 30 04:39:34.287695 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.287653    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-credential-socket\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.288309 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.288279    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-workload-socket\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.288441 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.288330    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-data\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.288571 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.288443    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-workload-certs\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.288571 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.288514    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frgxd\" (UniqueName: \"kubernetes.io/projected/17cf01ad-84ee-4f36-92f0-7cfff29f4215-kube-api-access-frgxd\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.288686 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.288578    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-podinfo\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.288686 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.288624    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istiod-ca-cert\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.288686 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.288672    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-token\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.288838 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.288704    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-envoy\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.390143 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.390106    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-credential-socket\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.390359 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.390191    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-workload-socket\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.390426 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.390363    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-data\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.390488 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.390462    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-workload-certs\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.390548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.390498    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frgxd\" (UniqueName: \"kubernetes.io/projected/17cf01ad-84ee-4f36-92f0-7cfff29f4215-kube-api-access-frgxd\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.390608 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.390555    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-podinfo\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.390666 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.390604    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istiod-ca-cert\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.390666 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.390607    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-workload-socket\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.390666 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.390650    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-credential-socket\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.390843 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.390673    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-token\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.390843 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.390750    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-envoy\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.390961 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.390917    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-data\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.391198 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.391175    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-workload-certs\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.391340 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.391301    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istiod-ca-cert\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.393020 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.392994    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-envoy\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.393368 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.393348    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-podinfo\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.422772 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.422752    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-token\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.423109 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.423088    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frgxd\" (UniqueName: \"kubernetes.io/projected/17cf01ad-84ee-4f36-92f0-7cfff29f4215-kube-api-access-frgxd\") pod \"e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") " pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.505056 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.505030    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:34.824654 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.824582    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"]
Oct 30 04:39:34.827724 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:39:34.827699    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17cf01ad_84ee_4f36_92f0_7cfff29f4215.slice/crio-2638a119f6ec44d73eb4bfb79e1fe17b8dcd5cbf38641e7f1fe0057f058e9e4c WatchSource:0}: Error finding container 2638a119f6ec44d73eb4bfb79e1fe17b8dcd5cbf38641e7f1fe0057f058e9e4c: Status 404 returned error can't find the container with id 2638a119f6ec44d73eb4bfb79e1fe17b8dcd5cbf38641e7f1fe0057f058e9e4c
Oct 30 04:39:34.830035 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.830007    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:39:34.830103 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.830070    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:39:34.830139 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:34.830105    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:39:35.292736 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:35.292700    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6" event={"ID":"17cf01ad-84ee-4f36-92f0-7cfff29f4215","Type":"ContainerStarted","Data":"b7b8d7e33783e6f609daf6a5979ea50cbe9f93708e0d823482a02b39d91ac504"}
Oct 30 04:39:35.293231 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:35.292758    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6" event={"ID":"17cf01ad-84ee-4f36-92f0-7cfff29f4215","Type":"ContainerStarted","Data":"2638a119f6ec44d73eb4bfb79e1fe17b8dcd5cbf38641e7f1fe0057f058e9e4c"}
Oct 30 04:39:35.336573 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:35.336517    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6" podStartSLOduration=1.336500856 podStartE2EDuration="1.336500856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:39:35.335843394 +0000 UTC m=+2494.748238970" watchObservedRunningTime="2026-10-30 04:39:35.336500856 +0000 UTC m=+2494.748896431"
Oct 30 04:39:35.505304 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:35.505274    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:35.510286 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:35.510252    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:35.712614 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:35.712535    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx"]
Oct 30 04:39:35.716282 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:35.716226    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx"
Oct 30 04:39:35.734371 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:35.734347    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-e070ea\""
Oct 30 04:39:35.734533 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:35.734491    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-e070ea-dockercfg-xzbsv\""
Oct 30 04:39:35.736922 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:35.736899    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx"]
Oct 30 04:39:35.806900 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:35.806856    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9be3939f-afda-4003-bd81-55e3349fdd86-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx\" (UID: \"9be3939f-afda-4003-bd81-55e3349fdd86\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx"
Oct 30 04:39:35.809865 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:35.807420    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fjk5\" (UniqueName: \"kubernetes.io/projected/9be3939f-afda-4003-bd81-55e3349fdd86-kube-api-access-4fjk5\") pod \"payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx\" (UID: \"9be3939f-afda-4003-bd81-55e3349fdd86\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx"
Oct 30 04:39:35.922531 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:35.922497    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fjk5\" (UniqueName: \"kubernetes.io/projected/9be3939f-afda-4003-bd81-55e3349fdd86-kube-api-access-4fjk5\") pod \"payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx\" (UID: \"9be3939f-afda-4003-bd81-55e3349fdd86\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx"
Oct 30 04:39:35.922681 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:35.922577    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9be3939f-afda-4003-bd81-55e3349fdd86-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx\" (UID: \"9be3939f-afda-4003-bd81-55e3349fdd86\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx"
Oct 30 04:39:35.923269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:35.923234    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9be3939f-afda-4003-bd81-55e3349fdd86-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx\" (UID: \"9be3939f-afda-4003-bd81-55e3349fdd86\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx"
Oct 30 04:39:35.947348 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:35.947322    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fjk5\" (UniqueName: \"kubernetes.io/projected/9be3939f-afda-4003-bd81-55e3349fdd86-kube-api-access-4fjk5\") pod \"payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx\" (UID: \"9be3939f-afda-4003-bd81-55e3349fdd86\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx"
Oct 30 04:39:36.030811 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:36.030787    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx"
Oct 30 04:39:36.297596 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:36.297520    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:36.298350 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:36.298329    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:39:36.359216 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:36.359187    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx"]
Oct 30 04:39:36.361932 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:39:36.361892    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be3939f_afda_4003_bd81_55e3349fdd86.slice/crio-befe2d885adab999e3332b6f9d9daac492f2f93a2435f74514999eeb034793c9 WatchSource:0}: Error finding container befe2d885adab999e3332b6f9d9daac492f2f93a2435f74514999eeb034793c9: Status 404 returned error can't find the container with id befe2d885adab999e3332b6f9d9daac492f2f93a2435f74514999eeb034793c9
Oct 30 04:39:37.304900 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:37.304865    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx" event={"ID":"9be3939f-afda-4003-bd81-55e3349fdd86","Type":"ContainerStarted","Data":"238f5d8804dd5366de753024615f9b4783e452d8cbbf983397116663c3e742ce"}
Oct 30 04:39:37.305330 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:37.304916    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx" event={"ID":"9be3939f-afda-4003-bd81-55e3349fdd86","Type":"ContainerStarted","Data":"befe2d885adab999e3332b6f9d9daac492f2f93a2435f74514999eeb034793c9"}
Oct 30 04:39:37.305330 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:37.305064    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx"
Oct 30 04:39:37.344060 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:37.343984    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx" podStartSLOduration=2.34397008 podStartE2EDuration="2.34397008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:39:37.343506436 +0000 UTC m=+2496.755902009" watchObservedRunningTime="2026-10-30 04:39:37.34397008 +0000 UTC m=+2496.756365654"
Oct 30 04:39:45.011721 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:45.011684    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz"]
Oct 30 04:39:45.015401 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:45.015384    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz"
Oct 30 04:39:45.035478 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:45.035451    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz"]
Oct 30 04:39:45.041546 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:45.041524    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-e6b837\""
Oct 30 04:39:45.041642 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:45.041541    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-e6b837-dockercfg-lfzwh\""
Oct 30 04:39:45.125005 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:45.124960    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwp8d\" (UniqueName: \"kubernetes.io/projected/beb04a4a-dc5c-4583-a464-3c56374faa33-kube-api-access-cwp8d\") pod \"payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz\" (UID: \"beb04a4a-dc5c-4583-a464-3c56374faa33\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz"
Oct 30 04:39:45.125191 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:45.125136    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/beb04a4a-dc5c-4583-a464-3c56374faa33-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz\" (UID: \"beb04a4a-dc5c-4583-a464-3c56374faa33\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz"
Oct 30 04:39:45.125493 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:45.125473    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb"]
Oct 30 04:39:45.130088 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:45.130066    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb"
Oct 30 04:39:45.164414 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:45.163844    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb"]
Oct 30 04:39:45.226659 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:45.226624    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653f8461-4436-4506-bc3e-0271173d425e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb\" (UID: \"653f8461-4436-4506-bc3e-0271173d425e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb"
Oct 30 04:39:45.226841 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:45.226698    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwp8d\" (UniqueName: \"kubernetes.io/projected/beb04a4a-dc5c-4583-a464-3c56374faa33-kube-api-access-cwp8d\") pod \"payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz\" (UID: \"beb04a4a-dc5c-4583-a464-3c56374faa33\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz"
Oct 30 04:39:45.226910 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:45.226856    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/beb04a4a-dc5c-4583-a464-3c56374faa33-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz\" (UID: \"beb04a4a-dc5c-4583-a464-3c56374faa33\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz"
Oct 30 04:39:45.226910 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:45.226898    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd7z6\" (UniqueName: \"kubernetes.io/projected/653f8461-4436-4506-bc3e-0271173d425e-kube-api-access-dd7z6\") pod \"payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb\" (UID: \"653f8461-4436-4506-bc3e-0271173d425e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb"
Oct 30 04:39:45.227549 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:45.227531    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/beb04a4a-dc5c-4583-a464-3c56374faa33-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz\" (UID: \"beb04a4a-dc5c-4583-a464-3c56374faa33\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz"
Oct 30 04:39:45.248708 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:45.248686    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwp8d\" (UniqueName: \"kubernetes.io/projected/beb04a4a-dc5c-4583-a464-3c56374faa33-kube-api-access-cwp8d\") pod \"payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz\" (UID: \"beb04a4a-dc5c-4583-a464-3c56374faa33\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz"
Oct 30 04:39:45.327752 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:45.327678    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd7z6\" (UniqueName: \"kubernetes.io/projected/653f8461-4436-4506-bc3e-0271173d425e-kube-api-access-dd7z6\") pod \"payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb\" (UID: \"653f8461-4436-4506-bc3e-0271173d425e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb"
Oct 30 04:39:45.327752 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:45.327732    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653f8461-4436-4506-bc3e-0271173d425e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb\" (UID: \"653f8461-4436-4506-bc3e-0271173d425e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb"
Oct 30 04:39:45.327931 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:45.327795    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz"
Oct 30 04:39:45.328488 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:45.328470    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653f8461-4436-4506-bc3e-0271173d425e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb\" (UID: \"653f8461-4436-4506-bc3e-0271173d425e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb"
Oct 30 04:39:45.350373 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:45.350347    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd7z6\" (UniqueName: \"kubernetes.io/projected/653f8461-4436-4506-bc3e-0271173d425e-kube-api-access-dd7z6\") pod \"payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb\" (UID: \"653f8461-4436-4506-bc3e-0271173d425e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb"
Oct 30 04:39:45.505359 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:45.505327    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb"
Oct 30 04:39:45.632270 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:45.632171    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz"]
Oct 30 04:39:45.635112 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:39:45.635081    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb04a4a_dc5c_4583_a464_3c56374faa33.slice/crio-fc1a3388d5a5e2dc3c9701d9213386ccd230723db7c5a03da72bc1a69b7fc939 WatchSource:0}: Error finding container fc1a3388d5a5e2dc3c9701d9213386ccd230723db7c5a03da72bc1a69b7fc939: Status 404 returned error can't find the container with id fc1a3388d5a5e2dc3c9701d9213386ccd230723db7c5a03da72bc1a69b7fc939
Oct 30 04:39:45.805316 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:45.805288    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb"]
Oct 30 04:39:45.808342 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:39:45.808320    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653f8461_4436_4506_bc3e_0271173d425e.slice/crio-3c5b9226c26d660050a4c640c3d72ece18f27a916e53c23b54bca3581b2675f0 WatchSource:0}: Error finding container 3c5b9226c26d660050a4c640c3d72ece18f27a916e53c23b54bca3581b2675f0: Status 404 returned error can't find the container with id 3c5b9226c26d660050a4c640c3d72ece18f27a916e53c23b54bca3581b2675f0
Oct 30 04:39:46.363966 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:46.363922    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz" event={"ID":"beb04a4a-dc5c-4583-a464-3c56374faa33","Type":"ContainerStarted","Data":"cd59cb01d7f88ed626c28acef56532c26568abd7ab4c04189f30a91ec706e324"}
Oct 30 04:39:46.364522 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:46.364019    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz" event={"ID":"beb04a4a-dc5c-4583-a464-3c56374faa33","Type":"ContainerStarted","Data":"fc1a3388d5a5e2dc3c9701d9213386ccd230723db7c5a03da72bc1a69b7fc939"}
Oct 30 04:39:46.364522 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:46.364051    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz"
Oct 30 04:39:46.365551 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:46.365523    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb" event={"ID":"653f8461-4436-4506-bc3e-0271173d425e","Type":"ContainerStarted","Data":"5be2d8adc28c67f08f9e0ec56639112b0f04c4e4cb9132c4c8c85e30ac1d33fb"}
Oct 30 04:39:46.365671 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:46.365561    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb" event={"ID":"653f8461-4436-4506-bc3e-0271173d425e","Type":"ContainerStarted","Data":"3c5b9226c26d660050a4c640c3d72ece18f27a916e53c23b54bca3581b2675f0"}
Oct 30 04:39:46.365671 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:46.365647    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb"
Oct 30 04:39:46.402532 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:46.402489    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz" podStartSLOduration=2.402475219 podStartE2EDuration="2.402475219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:39:46.401986619 +0000 UTC m=+2505.814382191" watchObservedRunningTime="2026-10-30 04:39:46.402475219 +0000 UTC m=+2505.814870794"
Oct 30 04:39:46.440420 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:46.440379    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb" podStartSLOduration=1.440367413 podStartE2EDuration="1.440367413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:39:46.439871909 +0000 UTC m=+2505.852267481" watchObservedRunningTime="2026-10-30 04:39:46.440367413 +0000 UTC m=+2505.852762986"
Oct 30 04:39:48.312893 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:48.312864    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx"
Oct 30 04:39:51.845900 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:51.845864    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h"]
Oct 30 04:39:51.849793 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:51.849771    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h"
Oct 30 04:39:51.863934 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:51.863918    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-66217b-serving-cert\""
Oct 30 04:39:51.878855 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:51.878827    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h"]
Oct 30 04:39:51.999881 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:51.999850    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4c5e053-08b5-4351-acf8-2e57775a9366-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h\" (UID: \"b4c5e053-08b5-4351-acf8-2e57775a9366\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h"
Oct 30 04:39:52.000049 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:51.999906    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4c5e053-08b5-4351-acf8-2e57775a9366-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h\" (UID: \"b4c5e053-08b5-4351-acf8-2e57775a9366\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h"
Oct 30 04:39:52.000049 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:51.999949    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljvgk\" (UniqueName: \"kubernetes.io/projected/b4c5e053-08b5-4351-acf8-2e57775a9366-kube-api-access-ljvgk\") pod \"maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h\" (UID: \"b4c5e053-08b5-4351-acf8-2e57775a9366\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h"
Oct 30 04:39:52.000124 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:52.000053    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4c5e053-08b5-4351-acf8-2e57775a9366-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h\" (UID: \"b4c5e053-08b5-4351-acf8-2e57775a9366\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h"
Oct 30 04:39:52.107308 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:52.104745    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4c5e053-08b5-4351-acf8-2e57775a9366-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h\" (UID: \"b4c5e053-08b5-4351-acf8-2e57775a9366\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h"
Oct 30 04:39:52.107308 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:52.104901    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4c5e053-08b5-4351-acf8-2e57775a9366-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h\" (UID: \"b4c5e053-08b5-4351-acf8-2e57775a9366\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h"
Oct 30 04:39:52.107308 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:52.104981    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4c5e053-08b5-4351-acf8-2e57775a9366-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h\" (UID: \"b4c5e053-08b5-4351-acf8-2e57775a9366\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h"
Oct 30 04:39:52.107308 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:52.105010    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljvgk\" (UniqueName: \"kubernetes.io/projected/b4c5e053-08b5-4351-acf8-2e57775a9366-kube-api-access-ljvgk\") pod \"maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h\" (UID: \"b4c5e053-08b5-4351-acf8-2e57775a9366\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h"
Oct 30 04:39:52.107308 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:52.106391    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4c5e053-08b5-4351-acf8-2e57775a9366-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h\" (UID: \"b4c5e053-08b5-4351-acf8-2e57775a9366\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h"
Oct 30 04:39:52.145666 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:52.145620    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4c5e053-08b5-4351-acf8-2e57775a9366-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h\" (UID: \"b4c5e053-08b5-4351-acf8-2e57775a9366\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h"
Oct 30 04:39:52.145861 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:52.145696    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4c5e053-08b5-4351-acf8-2e57775a9366-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h\" (UID: \"b4c5e053-08b5-4351-acf8-2e57775a9366\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h"
Oct 30 04:39:52.152258 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:52.150051    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljvgk\" (UniqueName: \"kubernetes.io/projected/b4c5e053-08b5-4351-acf8-2e57775a9366-kube-api-access-ljvgk\") pod \"maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h\" (UID: \"b4c5e053-08b5-4351-acf8-2e57775a9366\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h"
Oct 30 04:39:52.163570 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:52.163539    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h"
Oct 30 04:39:52.464560 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:52.464535    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h"]
Oct 30 04:39:52.467262 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:39:52.467222    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c5e053_08b5_4351_acf8_2e57775a9366.slice/crio-f342d6e1a0bd825451c283054df36cc8466a8281937fa2fa30aa9cd3a514a5da WatchSource:0}: Error finding container f342d6e1a0bd825451c283054df36cc8466a8281937fa2fa30aa9cd3a514a5da: Status 404 returned error can't find the container with id f342d6e1a0bd825451c283054df36cc8466a8281937fa2fa30aa9cd3a514a5da
Oct 30 04:39:52.468652 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:52.468635    4874 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 04:39:53.401479 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:53.401444    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h" event={"ID":"b4c5e053-08b5-4351-acf8-2e57775a9366","Type":"ContainerStarted","Data":"2d4383550b518314b37a9c985062e4afa067cd8eca840f8d7b472be7f7b0f49d"}
Oct 30 04:39:53.401882 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:53.401513    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h"
Oct 30 04:39:53.401882 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:53.401538    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h" event={"ID":"b4c5e053-08b5-4351-acf8-2e57775a9366","Type":"ContainerStarted","Data":"f342d6e1a0bd825451c283054df36cc8466a8281937fa2fa30aa9cd3a514a5da"}
Oct 30 04:39:53.442353 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:53.442294    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h" podStartSLOduration=2.141169481 podStartE2EDuration="2.442272083s" totalImagesPullingTime="301.102602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:39:53.441210352 +0000 UTC m=+2512.853605925" watchObservedRunningTime="2026-10-30 04:39:53.442272083 +0000 UTC m=+2512.854667653"
Oct 30 04:39:55.608960 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:39:55.608924    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:39:57.373400 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:57.373367    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb"
Oct 30 04:39:57.373789 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:57.373441    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz"
Oct 30 04:39:58.647499 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:58.647458    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:39:59.412927 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:39:59.412892    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h"
Oct 30 04:40:09.522566 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:09.522533    4874 generic.go:356] "Generic (PLEG): container finished" podID="833c472c-0226-4e47-bc11-5d2659845a86" containerID="ca61199c97394dfa8d2e03da7988c0080ccbbee9721005442a3ca2dbe62eb7c8" exitCode=0
Oct 30 04:40:09.522977 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:09.522606    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs" event={"ID":"833c472c-0226-4e47-bc11-5d2659845a86","Type":"ContainerDied","Data":"ca61199c97394dfa8d2e03da7988c0080ccbbee9721005442a3ca2dbe62eb7c8"}
Oct 30 04:40:09.523023 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:09.523000    4874 scope.go:122] "RemoveContainer" containerID="ca61199c97394dfa8d2e03da7988c0080ccbbee9721005442a3ca2dbe62eb7c8"
Oct 30 04:40:09.524351 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:09.524327    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:40:09.524402 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:09.524389    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:40:09.524441 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:09.524422    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:40:10.100920 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.100884    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"]
Oct 30 04:40:10.103590 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.103568    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.125189 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.125158    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-495ebe-openshift-default-dockercfg-76kt2\""
Oct 30 04:40:10.133338 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.133232    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"]
Oct 30 04:40:10.144620 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.144593    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-istio-envoy\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.144745 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.144664    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k589t\" (UniqueName: \"kubernetes.io/projected/d5587cde-1301-458b-80c5-5a83682b3e21-kube-api-access-k589t\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.144791 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.144745    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-workload-certs\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.144791 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.144778    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5587cde-1301-458b-80c5-5a83682b3e21-istio-podinfo\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.144881 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.144804    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-istio-data\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.144925 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.144900    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5587cde-1301-458b-80c5-5a83682b3e21-istio-token\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.144962 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.144936    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-credential-socket\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.144962 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.144956    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5587cde-1301-458b-80c5-5a83682b3e21-istiod-ca-cert\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.145083 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.145062    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-workload-socket\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.246077 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.246048    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-workload-socket\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.246223 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.246107    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-istio-envoy\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.246223 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.246140    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k589t\" (UniqueName: \"kubernetes.io/projected/d5587cde-1301-458b-80c5-5a83682b3e21-kube-api-access-k589t\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.246223 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.246172    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-workload-certs\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.246223 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.246193    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5587cde-1301-458b-80c5-5a83682b3e21-istio-podinfo\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.246469 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.246356    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-istio-data\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.246469 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.246447    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5587cde-1301-458b-80c5-5a83682b3e21-istio-token\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.246581 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.246498    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-credential-socket\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.246581 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.246526    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5587cde-1301-458b-80c5-5a83682b3e21-istiod-ca-cert\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.246581 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.246554    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-workload-socket\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.246751 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.246594    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-workload-certs\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.246751 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.246707    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-istio-data\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.246837 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.246773    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-credential-socket\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.247097 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.247071    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5587cde-1301-458b-80c5-5a83682b3e21-istiod-ca-cert\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.248369 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.248349    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-istio-envoy\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.248763 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.248744    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5587cde-1301-458b-80c5-5a83682b3e21-istio-podinfo\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.268575 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.268553    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5587cde-1301-458b-80c5-5a83682b3e21-istio-token\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.269079 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.269056    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k589t\" (UniqueName: \"kubernetes.io/projected/d5587cde-1301-458b-80c5-5a83682b3e21-kube-api-access-k589t\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.416810 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.416734    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:10.531052 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.531020    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs" event={"ID":"833c472c-0226-4e47-bc11-5d2659845a86","Type":"ContainerStarted","Data":"35c276127baf4f367c64e15b4614c9c08c0d99686ea82bceded1649a17163ee4"}
Oct 30 04:40:10.780618 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.780579    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"]
Oct 30 04:40:10.791267 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:40:10.791183    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5587cde_1301_458b_80c5_5a83682b3e21.slice/crio-9a2d51d55435a2285a44e0ea3c4b88e8bc097dd48ede42ac4e7f32b549ddcfac WatchSource:0}: Error finding container 9a2d51d55435a2285a44e0ea3c4b88e8bc097dd48ede42ac4e7f32b549ddcfac: Status 404 returned error can't find the container with id 9a2d51d55435a2285a44e0ea3c4b88e8bc097dd48ede42ac4e7f32b549ddcfac
Oct 30 04:40:10.794203 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.794167    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:40:10.794360 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.794293    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:40:10.794360 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:10.794339    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:40:11.524729 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:11.524694    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:40:11.524920 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:11.524893    4874 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 30 04:40:11.525019 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:11.524952    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs" podUID="833c472c-0226-4e47-bc11-5d2659845a86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 30 04:40:11.538208 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:11.538178    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h" event={"ID":"d5587cde-1301-458b-80c5-5a83682b3e21","Type":"ContainerStarted","Data":"94a89367cef3a0bb978ace6f7734f152e213169061d6bf73441c286f51180167"}
Oct 30 04:40:11.538587 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:11.538229    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h" event={"ID":"d5587cde-1301-458b-80c5-5a83682b3e21","Type":"ContainerStarted","Data":"9a2d51d55435a2285a44e0ea3c4b88e8bc097dd48ede42ac4e7f32b549ddcfac"}
Oct 30 04:40:11.590838 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:11.590774    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h" podStartSLOduration=1.590752868 podStartE2EDuration="1.590752868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:40:11.583639817 +0000 UTC m=+2530.996035415" watchObservedRunningTime="2026-10-30 04:40:11.590752868 +0000 UTC m=+2531.003148441"
Oct 30 04:40:12.417266 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:12.417203    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:12.422825 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:12.422798    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:12.524617 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:12.524580    4874 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 30 04:40:12.524814 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:12.524643    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs" podUID="833c472c-0226-4e47-bc11-5d2659845a86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 30 04:40:12.544019 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:12.543992    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:12.544864 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:12.544838    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:13.525570 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:13.525528    4874 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 30 04:40:13.525775 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:13.525594    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs" podUID="833c472c-0226-4e47-bc11-5d2659845a86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 30 04:40:14.524659 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:14.524624    4874 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 30 04:40:14.525070 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:14.524679    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs" podUID="833c472c-0226-4e47-bc11-5d2659845a86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 30 04:40:15.524799 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:15.524764    4874 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 30 04:40:15.525264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:15.524815    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs" podUID="833c472c-0226-4e47-bc11-5d2659845a86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 30 04:40:16.528784 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:16.528754    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:40:16.529218 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:16.529033    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:40:16.529730 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:16.529712    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 04:40:22.339376 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:22.339337    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 04:40:22.339817 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:22.339397    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6" podUID="17cf01ad-84ee-4f36-92f0-7cfff29f4215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 04:40:22.931982 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:22.931914    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"]
Oct 30 04:40:22.932172 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:22.932117    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4" podUID="81d0f966-46ee-4f8a-9caa-97c985bcd2a8" containerName="istio-proxy" containerID="cri-o://37a6ca78766e2559ee222922676d8733b32b08173c5297c97e18668b41036787" gracePeriod=30
Oct 30 04:40:23.636589 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:23.636555    4874 generic.go:356] "Generic (PLEG): container finished" podID="17cf01ad-84ee-4f36-92f0-7cfff29f4215" containerID="b7b8d7e33783e6f609daf6a5979ea50cbe9f93708e0d823482a02b39d91ac504" exitCode=0
Oct 30 04:40:23.636589 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:23.636596    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6" event={"ID":"17cf01ad-84ee-4f36-92f0-7cfff29f4215","Type":"ContainerDied","Data":"b7b8d7e33783e6f609daf6a5979ea50cbe9f93708e0d823482a02b39d91ac504"}
Oct 30 04:40:23.637016 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:23.636950    4874 scope.go:122] "RemoveContainer" containerID="b7b8d7e33783e6f609daf6a5979ea50cbe9f93708e0d823482a02b39d91ac504"
Oct 30 04:40:23.638365 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:23.638334    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:40:23.638453 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:23.638409    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:40:23.638453 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:23.638442    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:40:24.645789 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:24.645755    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6" event={"ID":"17cf01ad-84ee-4f36-92f0-7cfff29f4215","Type":"ContainerStarted","Data":"cfd5317af1b4f6feba4f5881d93d8ee9f04554f05c57ce4a25ad7a6954d8f42f"}
Oct 30 04:40:25.054347 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:25.054311    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:40:25.505853 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:25.505814    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:40:25.506148 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:25.506123    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 04:40:25.506221 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:25.506192    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6" podUID="17cf01ad-84ee-4f36-92f0-7cfff29f4215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 04:40:25.617203 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:40:25.617170    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:40:25.894487 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:25.894397    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-x8gd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 04:40:25.895054 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:25.895021    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4" podUID="81d0f966-46ee-4f8a-9caa-97c985bcd2a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:40:26.506324 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:26.506271    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 04:40:26.506541 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:26.506344    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6" podUID="17cf01ad-84ee-4f36-92f0-7cfff29f4215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 04:40:26.587264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:26.587142    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"]
Oct 30 04:40:26.657716 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:26.657680    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc" podUID="b488e852-f9e1-49ea-ac86-b3f409299806" containerName="istio-proxy" containerID="cri-o://b91fd174552fbc3629dea60259187237f2ff95ff8f678ddb4b0fdff2775455cb" gracePeriod=30
Oct 30 04:40:27.505855 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:27.505818    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 04:40:27.506273 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:27.505881    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6" podUID="17cf01ad-84ee-4f36-92f0-7cfff29f4215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 04:40:28.226405 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.226380    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:40:28.259860 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.259832    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-kube-api-access-sf576\" (UniqueName: \"kubernetes.io/projected/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-kube-api-access-sf576\") pod \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") "
Oct 30 04:40:28.259985 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.259879    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-workload-socket\") pod \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") "
Oct 30 04:40:28.259985 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.259934    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-token\" (UniqueName: \"kubernetes.io/projected/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-token\") pod \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") "
Oct 30 04:40:28.259985 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.259967    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-credential-socket\") pod \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") "
Oct 30 04:40:28.260122 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.260036    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-workload-certs\") pod \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") "
Oct 30 04:40:28.260122 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.260101    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-data\") pod \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") "
Oct 30 04:40:28.260212 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.260170    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istiod-ca-cert\") pod \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") "
Oct 30 04:40:28.260305 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.260215    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-envoy\") pod \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") "
Oct 30 04:40:28.260305 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.260232    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-workload-socket" pod "81d0f966-46ee-4f8a-9caa-97c985bcd2a8" (UID: "81d0f966-46ee-4f8a-9caa-97c985bcd2a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:40:28.260305 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.260267    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-podinfo\") pod \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\" (UID: \"81d0f966-46ee-4f8a-9caa-97c985bcd2a8\") "
Oct 30 04:40:28.260471 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.260309    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-credential-socket" pod "81d0f966-46ee-4f8a-9caa-97c985bcd2a8" (UID: "81d0f966-46ee-4f8a-9caa-97c985bcd2a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:40:28.260471 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.260366    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-workload-certs" pod "81d0f966-46ee-4f8a-9caa-97c985bcd2a8" (UID: "81d0f966-46ee-4f8a-9caa-97c985bcd2a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:40:28.260606 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.260578    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-data" pod "81d0f966-46ee-4f8a-9caa-97c985bcd2a8" (UID: "81d0f966-46ee-4f8a-9caa-97c985bcd2a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:40:28.260778 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.260752    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istiod-ca-cert" pod "81d0f966-46ee-4f8a-9caa-97c985bcd2a8" (UID: "81d0f966-46ee-4f8a-9caa-97c985bcd2a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:40:28.260937 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.260916    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:28.260937 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.260940    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:28.261079 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.260954    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:28.261079 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.260967    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:28.261079 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.260980    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:28.262161 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.262137    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-kube-api-access-sf576" pod "81d0f966-46ee-4f8a-9caa-97c985bcd2a8" (UID: "81d0f966-46ee-4f8a-9caa-97c985bcd2a8"). InnerVolumeSpecName "kube-api-access-sf576". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:40:28.262462 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.262441    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-token" pod "81d0f966-46ee-4f8a-9caa-97c985bcd2a8" (UID: "81d0f966-46ee-4f8a-9caa-97c985bcd2a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:40:28.262527 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.262482    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-envoy" pod "81d0f966-46ee-4f8a-9caa-97c985bcd2a8" (UID: "81d0f966-46ee-4f8a-9caa-97c985bcd2a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:40:28.262527 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.262489    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-podinfo" pod "81d0f966-46ee-4f8a-9caa-97c985bcd2a8" (UID: "81d0f966-46ee-4f8a-9caa-97c985bcd2a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:40:28.361811 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.361783    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:28.361811 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.361807    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:28.361811 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.361816    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:28.362011 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.361825    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf576\" (UniqueName: \"kubernetes.io/projected/81d0f966-46ee-4f8a-9caa-97c985bcd2a8-kube-api-access-sf576\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:28.506323 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.506289    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 04:40:28.506689 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.506349    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6" podUID="17cf01ad-84ee-4f36-92f0-7cfff29f4215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 04:40:28.690358 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.690319    4874 generic.go:356] "Generic (PLEG): container finished" podID="81d0f966-46ee-4f8a-9caa-97c985bcd2a8" containerID="37a6ca78766e2559ee222922676d8733b32b08173c5297c97e18668b41036787" exitCode=0
Oct 30 04:40:28.690358 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.690363    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"
Oct 30 04:40:28.690625 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.690413    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4" event={"ID":"81d0f966-46ee-4f8a-9caa-97c985bcd2a8","Type":"ContainerDied","Data":"37a6ca78766e2559ee222922676d8733b32b08173c5297c97e18668b41036787"}
Oct 30 04:40:28.690625 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.690475    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4" event={"ID":"81d0f966-46ee-4f8a-9caa-97c985bcd2a8","Type":"ContainerDied","Data":"aefc16e66959da51a8d9f9df0987d8accd8b1b0225cb25e2109118040257e062"}
Oct 30 04:40:28.690625 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.690508    4874 scope.go:122] "RemoveContainer" containerID="37a6ca78766e2559ee222922676d8733b32b08173c5297c97e18668b41036787"
Oct 30 04:40:28.702968 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.702949    4874 scope.go:122] "RemoveContainer" containerID="37a6ca78766e2559ee222922676d8733b32b08173c5297c97e18668b41036787"
Oct 30 04:40:28.703261 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:40:28.703227    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37a6ca78766e2559ee222922676d8733b32b08173c5297c97e18668b41036787\": container with ID starting with 37a6ca78766e2559ee222922676d8733b32b08173c5297c97e18668b41036787 not found: ID does not exist" containerID="37a6ca78766e2559ee222922676d8733b32b08173c5297c97e18668b41036787"
Oct 30 04:40:28.703332 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.703269    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37a6ca78766e2559ee222922676d8733b32b08173c5297c97e18668b41036787"} err="failed to get container status \"37a6ca78766e2559ee222922676d8733b32b08173c5297c97e18668b41036787\": rpc error: code = NotFound desc = could not find container \"37a6ca78766e2559ee222922676d8733b32b08173c5297c97e18668b41036787\": container with ID starting with 37a6ca78766e2559ee222922676d8733b32b08173c5297c97e18668b41036787 not found: ID does not exist"
Oct 30 04:40:28.745211 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.745189    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"]
Oct 30 04:40:28.759083 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:28.759000    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8gd4"]
Oct 30 04:40:29.505812 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:29.505777    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 04:40:29.505984 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:29.505831    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6" podUID="17cf01ad-84ee-4f36-92f0-7cfff29f4215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 04:40:30.509810 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.509780    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:40:30.510192 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.510069    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:40:30.510797 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.510776    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 04:40:30.515717 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.515699    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj"]
Oct 30 04:40:30.516150 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.516138    4874 cpu_manager.go:414] "containerMap: removing container" podUID="81d0f966-46ee-4f8a-9caa-97c985bcd2a8" containerName="istio-proxy"
Oct 30 04:40:30.516188 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.516151    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81d0f966-46ee-4f8a-9caa-97c985bcd2a8" containerName="istio-proxy"
Oct 30 04:40:30.516261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.516228    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="81d0f966-46ee-4f8a-9caa-97c985bcd2a8" containerName="istio-proxy"
Oct 30 04:40:30.518252 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.518225    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj"
Oct 30 04:40:30.544178 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.544157    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj"]
Oct 30 04:40:30.652403 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.649188    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-247158-dockercfg-6dqtk\""
Oct 30 04:40:30.657169 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.655083    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-247158\""
Oct 30 04:40:30.657380 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.657075    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e900ea6-5bdb-40c7-bcd2-de0a6910aac0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj\" (UID: \"6e900ea6-5bdb-40c7-bcd2-de0a6910aac0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj"
Oct 30 04:40:30.658270 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.657655    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j5g7\" (UniqueName: \"kubernetes.io/projected/6e900ea6-5bdb-40c7-bcd2-de0a6910aac0-kube-api-access-8j5g7\") pod \"payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj\" (UID: \"6e900ea6-5bdb-40c7-bcd2-de0a6910aac0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj"
Oct 30 04:40:30.658431 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.658346    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s"]
Oct 30 04:40:30.662659 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.662640    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s"]
Oct 30 04:40:30.662910 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.662850    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s"
Oct 30 04:40:30.701467 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.701437    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-495ebe-serving-cert\""
Oct 30 04:40:30.730470 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.730441    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81d0f966-46ee-4f8a-9caa-97c985bcd2a8" path="/var/lib/kubelet/pods/81d0f966-46ee-4f8a-9caa-97c985bcd2a8/volumes"
Oct 30 04:40:30.759807 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.759779    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e05329a8-9320-47fe-9766-622ae3c93bc4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s\" (UID: \"e05329a8-9320-47fe-9766-622ae3c93bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s"
Oct 30 04:40:30.759983 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.759836    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j5g7\" (UniqueName: \"kubernetes.io/projected/6e900ea6-5bdb-40c7-bcd2-de0a6910aac0-kube-api-access-8j5g7\") pod \"payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj\" (UID: \"6e900ea6-5bdb-40c7-bcd2-de0a6910aac0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj"
Oct 30 04:40:30.759983 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.759892    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2vgv\" (UniqueName: \"kubernetes.io/projected/e05329a8-9320-47fe-9766-622ae3c93bc4-kube-api-access-r2vgv\") pod \"maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s\" (UID: \"e05329a8-9320-47fe-9766-622ae3c93bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s"
Oct 30 04:40:30.759983 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.759941    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e05329a8-9320-47fe-9766-622ae3c93bc4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s\" (UID: \"e05329a8-9320-47fe-9766-622ae3c93bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s"
Oct 30 04:40:30.760306 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.760287    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e05329a8-9320-47fe-9766-622ae3c93bc4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s\" (UID: \"e05329a8-9320-47fe-9766-622ae3c93bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s"
Oct 30 04:40:30.760377 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.760334    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e900ea6-5bdb-40c7-bcd2-de0a6910aac0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj\" (UID: \"6e900ea6-5bdb-40c7-bcd2-de0a6910aac0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj"
Oct 30 04:40:30.761274 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.761229    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e900ea6-5bdb-40c7-bcd2-de0a6910aac0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj\" (UID: \"6e900ea6-5bdb-40c7-bcd2-de0a6910aac0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj"
Oct 30 04:40:30.782660 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.782629    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j5g7\" (UniqueName: \"kubernetes.io/projected/6e900ea6-5bdb-40c7-bcd2-de0a6910aac0-kube-api-access-8j5g7\") pod \"payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj\" (UID: \"6e900ea6-5bdb-40c7-bcd2-de0a6910aac0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj"
Oct 30 04:40:30.847260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.846502    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj"
Oct 30 04:40:30.861461 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.861422    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e05329a8-9320-47fe-9766-622ae3c93bc4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s\" (UID: \"e05329a8-9320-47fe-9766-622ae3c93bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s"
Oct 30 04:40:30.861610 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.861520    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2vgv\" (UniqueName: \"kubernetes.io/projected/e05329a8-9320-47fe-9766-622ae3c93bc4-kube-api-access-r2vgv\") pod \"maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s\" (UID: \"e05329a8-9320-47fe-9766-622ae3c93bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s"
Oct 30 04:40:30.861610 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.861556    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e05329a8-9320-47fe-9766-622ae3c93bc4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s\" (UID: \"e05329a8-9320-47fe-9766-622ae3c93bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s"
Oct 30 04:40:30.861767 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.861638    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e05329a8-9320-47fe-9766-622ae3c93bc4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s\" (UID: \"e05329a8-9320-47fe-9766-622ae3c93bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s"
Oct 30 04:40:30.862526 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.862490    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e05329a8-9320-47fe-9766-622ae3c93bc4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s\" (UID: \"e05329a8-9320-47fe-9766-622ae3c93bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s"
Oct 30 04:40:30.863937 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.863915    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e05329a8-9320-47fe-9766-622ae3c93bc4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s\" (UID: \"e05329a8-9320-47fe-9766-622ae3c93bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s"
Oct 30 04:40:30.864467 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.864445    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e05329a8-9320-47fe-9766-622ae3c93bc4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s\" (UID: \"e05329a8-9320-47fe-9766-622ae3c93bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s"
Oct 30 04:40:30.884531 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.884068    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2vgv\" (UniqueName: \"kubernetes.io/projected/e05329a8-9320-47fe-9766-622ae3c93bc4-kube-api-access-r2vgv\") pod \"maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s\" (UID: \"e05329a8-9320-47fe-9766-622ae3c93bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s"
Oct 30 04:40:30.956020 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.955989    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"]
Oct 30 04:40:30.976724 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:30.976694    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s"
Oct 30 04:40:31.152738 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:31.152714    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj"]
Oct 30 04:40:31.155213 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:40:31.155180    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e900ea6_5bdb_40c7_bcd2_de0a6910aac0.slice/crio-ac8c8df7b4bf22fd8c53d070906151a7b792b2a1bef3822a985742556856db2c WatchSource:0}: Error finding container ac8c8df7b4bf22fd8c53d070906151a7b792b2a1bef3822a985742556856db2c: Status 404 returned error can't find the container with id ac8c8df7b4bf22fd8c53d070906151a7b792b2a1bef3822a985742556856db2c
Oct 30 04:40:31.276701 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:31.276656    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s"]
Oct 30 04:40:31.279249 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:40:31.279212    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode05329a8_9320_47fe_9766_622ae3c93bc4.slice/crio-9f731bfba0cdebccd0b01b12ae64a5e6e8d54f1eada489196afc81c3c619800d WatchSource:0}: Error finding container 9f731bfba0cdebccd0b01b12ae64a5e6e8d54f1eada489196afc81c3c619800d: Status 404 returned error can't find the container with id 9f731bfba0cdebccd0b01b12ae64a5e6e8d54f1eada489196afc81c3c619800d
Oct 30 04:40:31.711331 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:31.711297    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj" event={"ID":"6e900ea6-5bdb-40c7-bcd2-de0a6910aac0","Type":"ContainerStarted","Data":"6a93dc9501e984d7b9dfbb1b22493e35ce6fb6da3432a1e1e502c0d12deab77e"}
Oct 30 04:40:31.711719 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:31.711352    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj" event={"ID":"6e900ea6-5bdb-40c7-bcd2-de0a6910aac0","Type":"ContainerStarted","Data":"ac8c8df7b4bf22fd8c53d070906151a7b792b2a1bef3822a985742556856db2c"}
Oct 30 04:40:31.711719 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:31.711502    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj"
Oct 30 04:40:31.712653 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:31.712630    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s" event={"ID":"e05329a8-9320-47fe-9766-622ae3c93bc4","Type":"ContainerStarted","Data":"9f731bfba0cdebccd0b01b12ae64a5e6e8d54f1eada489196afc81c3c619800d"}
Oct 30 04:40:31.712758 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:31.712679    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq" podUID="b3e828ca-ccfa-471b-a2ee-4799f2edb09c" containerName="istio-proxy" containerID="cri-o://a84cc1a728f76def95c124aedbf8f320cba0899d25f57e7adc8129ea27fdaa94" gracePeriod=30
Oct 30 04:40:31.794271 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:31.791421    4874 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 04:40:31.794271 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:31.791476    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq" podUID="b3e828ca-ccfa-471b-a2ee-4799f2edb09c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:40:31.804793 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:31.803819    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj" podStartSLOduration=1.803801264 podStartE2EDuration="1.803801264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:40:31.759566418 +0000 UTC m=+2551.171961992" watchObservedRunningTime="2026-10-30 04:40:31.803801264 +0000 UTC m=+2551.216196838"
Oct 30 04:40:31.969191 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:31.969167    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:40:32.105147 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.105116    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b488e852-f9e1-49ea-ac86-b3f409299806-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b488e852-f9e1-49ea-ac86-b3f409299806-istiod-ca-cert\") pod \"b488e852-f9e1-49ea-ac86-b3f409299806\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") "
Oct 30 04:40:32.105317 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.105190    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-workload-socket\") pod \"b488e852-f9e1-49ea-ac86-b3f409299806\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") "
Oct 30 04:40:32.105379 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.105358    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b488e852-f9e1-49ea-ac86-b3f409299806-kube-api-access-fnqqs\" (UniqueName: \"kubernetes.io/projected/b488e852-f9e1-49ea-ac86-b3f409299806-kube-api-access-fnqqs\") pod \"b488e852-f9e1-49ea-ac86-b3f409299806\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") "
Oct 30 04:40:32.105451 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.105406    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-istio-envoy\") pod \"b488e852-f9e1-49ea-ac86-b3f409299806\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") "
Oct 30 04:40:32.105451 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.105440    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-credential-socket\") pod \"b488e852-f9e1-49ea-ac86-b3f409299806\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") "
Oct 30 04:40:32.105564 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.105459    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-workload-socket" pod "b488e852-f9e1-49ea-ac86-b3f409299806" (UID: "b488e852-f9e1-49ea-ac86-b3f409299806"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:40:32.105564 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.105493    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-workload-certs\") pod \"b488e852-f9e1-49ea-ac86-b3f409299806\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") "
Oct 30 04:40:32.105564 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.105496    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b488e852-f9e1-49ea-ac86-b3f409299806-istiod-ca-cert" pod "b488e852-f9e1-49ea-ac86-b3f409299806" (UID: "b488e852-f9e1-49ea-ac86-b3f409299806"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:40:32.105564 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.105532    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-istio-data\") pod \"b488e852-f9e1-49ea-ac86-b3f409299806\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") "
Oct 30 04:40:32.105756 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.105562    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b488e852-f9e1-49ea-ac86-b3f409299806-istio-token\" (UniqueName: \"kubernetes.io/projected/b488e852-f9e1-49ea-ac86-b3f409299806-istio-token\") pod \"b488e852-f9e1-49ea-ac86-b3f409299806\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") "
Oct 30 04:40:32.105756 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.105598    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b488e852-f9e1-49ea-ac86-b3f409299806-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b488e852-f9e1-49ea-ac86-b3f409299806-istio-podinfo\") pod \"b488e852-f9e1-49ea-ac86-b3f409299806\" (UID: \"b488e852-f9e1-49ea-ac86-b3f409299806\") "
Oct 30 04:40:32.105756 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.105704    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-credential-socket" pod "b488e852-f9e1-49ea-ac86-b3f409299806" (UID: "b488e852-f9e1-49ea-ac86-b3f409299806"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:40:32.105905 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.105801    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-workload-certs" pod "b488e852-f9e1-49ea-ac86-b3f409299806" (UID: "b488e852-f9e1-49ea-ac86-b3f409299806"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:40:32.106136 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.106118    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:32.106227 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.106140    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b488e852-f9e1-49ea-ac86-b3f409299806-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:32.106227 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.106138    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-istio-data" pod "b488e852-f9e1-49ea-ac86-b3f409299806" (UID: "b488e852-f9e1-49ea-ac86-b3f409299806"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:40:32.106227 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.106152    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:32.106227 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.106166    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:32.107737 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.107707    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b488e852-f9e1-49ea-ac86-b3f409299806-istio-podinfo" pod "b488e852-f9e1-49ea-ac86-b3f409299806" (UID: "b488e852-f9e1-49ea-ac86-b3f409299806"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:40:32.107978 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.107960    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b488e852-f9e1-49ea-ac86-b3f409299806-kube-api-access-fnqqs" pod "b488e852-f9e1-49ea-ac86-b3f409299806" (UID: "b488e852-f9e1-49ea-ac86-b3f409299806"). InnerVolumeSpecName "kube-api-access-fnqqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:40:32.108189 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.108163    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b488e852-f9e1-49ea-ac86-b3f409299806-istio-token" pod "b488e852-f9e1-49ea-ac86-b3f409299806" (UID: "b488e852-f9e1-49ea-ac86-b3f409299806"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:40:32.108189 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.108172    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-istio-envoy" pod "b488e852-f9e1-49ea-ac86-b3f409299806" (UID: "b488e852-f9e1-49ea-ac86-b3f409299806"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:40:32.207995 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.207810    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnqqs\" (UniqueName: \"kubernetes.io/projected/b488e852-f9e1-49ea-ac86-b3f409299806-kube-api-access-fnqqs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:32.207995 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.207852    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:32.207995 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.207869    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b488e852-f9e1-49ea-ac86-b3f409299806-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:32.207995 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.207880    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b488e852-f9e1-49ea-ac86-b3f409299806-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:32.207995 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.207892    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b488e852-f9e1-49ea-ac86-b3f409299806-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:32.720490 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.720454    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s" event={"ID":"e05329a8-9320-47fe-9766-622ae3c93bc4","Type":"ContainerStarted","Data":"0a15d423ce486758eedac184592c3107ea361774d842826cc213f06acb78c9c2"}
Oct 30 04:40:32.720980 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.720537    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s"
Oct 30 04:40:32.721859 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.721836    4874 generic.go:356] "Generic (PLEG): container finished" podID="b488e852-f9e1-49ea-ac86-b3f409299806" containerID="b91fd174552fbc3629dea60259187237f2ff95ff8f678ddb4b0fdff2775455cb" exitCode=0
Oct 30 04:40:32.721962 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.721941    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"
Oct 30 04:40:32.737841 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.737819    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc" event={"ID":"b488e852-f9e1-49ea-ac86-b3f409299806","Type":"ContainerDied","Data":"b91fd174552fbc3629dea60259187237f2ff95ff8f678ddb4b0fdff2775455cb"}
Oct 30 04:40:32.737926 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.737844    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc" event={"ID":"b488e852-f9e1-49ea-ac86-b3f409299806","Type":"ContainerDied","Data":"ffcaf87313b4a7ea4d01ffb802c0ba0d4b3cf1d0bf1d60c2e60fb73ae786ff83"}
Oct 30 04:40:32.737926 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.737863    4874 scope.go:122] "RemoveContainer" containerID="b91fd174552fbc3629dea60259187237f2ff95ff8f678ddb4b0fdff2775455cb"
Oct 30 04:40:32.747421 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.747387    4874 scope.go:122] "RemoveContainer" containerID="b91fd174552fbc3629dea60259187237f2ff95ff8f678ddb4b0fdff2775455cb"
Oct 30 04:40:32.747702 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:40:32.747682    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b91fd174552fbc3629dea60259187237f2ff95ff8f678ddb4b0fdff2775455cb\": container with ID starting with b91fd174552fbc3629dea60259187237f2ff95ff8f678ddb4b0fdff2775455cb not found: ID does not exist" containerID="b91fd174552fbc3629dea60259187237f2ff95ff8f678ddb4b0fdff2775455cb"
Oct 30 04:40:32.747783 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.747713    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b91fd174552fbc3629dea60259187237f2ff95ff8f678ddb4b0fdff2775455cb"} err="failed to get container status \"b91fd174552fbc3629dea60259187237f2ff95ff8f678ddb4b0fdff2775455cb\": rpc error: code = NotFound desc = could not find container \"b91fd174552fbc3629dea60259187237f2ff95ff8f678ddb4b0fdff2775455cb\": container with ID starting with b91fd174552fbc3629dea60259187237f2ff95ff8f678ddb4b0fdff2775455cb not found: ID does not exist"
Oct 30 04:40:32.762635 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.762594    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s" podStartSLOduration=2.406067987 podStartE2EDuration="2.76258271s" totalImagesPullingTime="356.514723ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:40:32.760989283 +0000 UTC m=+2552.173384857" watchObservedRunningTime="2026-10-30 04:40:32.76258271 +0000 UTC m=+2552.174978282"
Oct 30 04:40:32.795466 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.795443    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"]
Oct 30 04:40:32.800033 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:32.800008    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-9nshc"]
Oct 30 04:40:33.922080 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:33.922051    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"]
Oct 30 04:40:33.922583 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:33.922568    4874 cpu_manager.go:414] "containerMap: removing container" podUID="b488e852-f9e1-49ea-ac86-b3f409299806" containerName="istio-proxy"
Oct 30 04:40:33.922627 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:33.922583    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b488e852-f9e1-49ea-ac86-b3f409299806" containerName="istio-proxy"
Oct 30 04:40:33.922688 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:33.922679    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="b488e852-f9e1-49ea-ac86-b3f409299806" containerName="istio-proxy"
Oct 30 04:40:33.925948 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:33.925923    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:33.940821 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:33.940798    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-fcd6a6-openshift-default-dockercfg-n97h9\""
Oct 30 04:40:33.955019 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:33.954994    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"]
Oct 30 04:40:34.029630 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.029600    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-credential-socket\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.029757 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.029658    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-workload-socket\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.029757 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.029742    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-token\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.029902 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.029850    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-workload-certs\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.029958 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.029927    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-podinfo\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.030010 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.029959    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4sfd\" (UniqueName: \"kubernetes.io/projected/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-kube-api-access-j4sfd\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.030081 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.030064    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-data\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.030144 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.030107    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-envoy\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.030210 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.030186    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istiod-ca-cert\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.136501 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.136329    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-podinfo\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.136501 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.136386    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4sfd\" (UniqueName: \"kubernetes.io/projected/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-kube-api-access-j4sfd\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.136501 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.136432    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-data\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.137338 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.136898    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-data\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.137338 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.136972    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-envoy\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.137338 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.137017    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istiod-ca-cert\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.137338 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.137119    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-credential-socket\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.137338 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.137175    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-workload-socket\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.137338 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.137206    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-token\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.137743 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.137353    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-workload-certs\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.137743 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.137731    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-workload-certs\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.147988 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.147952    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-credential-socket\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.149477 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.149421    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-workload-socket\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.151183 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.151011    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istiod-ca-cert\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.157215 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.157015    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-envoy\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.162822 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.162798    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-podinfo\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.170395 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.170370    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4sfd\" (UniqueName: \"kubernetes.io/projected/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-kube-api-access-j4sfd\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.171027 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.171008    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-token\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.238091 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.238067    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:34.555594 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.555571    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"]
Oct 30 04:40:34.557258 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:40:34.557215    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89aeca9_c4fc_4f9c_ba74_033b72349ddf.slice/crio-28da3b91fb96e593651e59a4bf5e0e3aebb6ce7a769226a52f64a82a537890de WatchSource:0}: Error finding container 28da3b91fb96e593651e59a4bf5e0e3aebb6ce7a769226a52f64a82a537890de: Status 404 returned error can't find the container with id 28da3b91fb96e593651e59a4bf5e0e3aebb6ce7a769226a52f64a82a537890de
Oct 30 04:40:34.559674 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.559640    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:40:34.559750 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.559721    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:40:34.559783 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.559751    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:40:34.730306 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.730268    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b488e852-f9e1-49ea-ac86-b3f409299806" path="/var/lib/kubelet/pods/b488e852-f9e1-49ea-ac86-b3f409299806/volumes"
Oct 30 04:40:34.737683 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.737654    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p" event={"ID":"c89aeca9-c4fc-4f9c-ba74-033b72349ddf","Type":"ContainerStarted","Data":"5efff0f058281139942ad6c2ab4fd98d67181a178e1b3d8cd72658c74a1475f5"}
Oct 30 04:40:34.737828 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.737703    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p" event={"ID":"c89aeca9-c4fc-4f9c-ba74-033b72349ddf","Type":"ContainerStarted","Data":"28da3b91fb96e593651e59a4bf5e0e3aebb6ce7a769226a52f64a82a537890de"}
Oct 30 04:40:34.782593 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:34.782539    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p" podStartSLOduration=1.7825221230000001 podStartE2EDuration="1.782522123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:40:34.780538551 +0000 UTC m=+2554.192934123" watchObservedRunningTime="2026-10-30 04:40:34.782522123 +0000 UTC m=+2554.194917700"
Oct 30 04:40:35.238416 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:35.238382    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:35.240113 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:35.240086    4874 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 30 04:40:35.240297 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:35.240134    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p" podUID="c89aeca9-c4fc-4f9c-ba74-033b72349ddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 30 04:40:36.238504 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:36.238464    4874 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 30 04:40:36.238941 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:36.238531    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p" podUID="c89aeca9-c4fc-4f9c-ba74-033b72349ddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 30 04:40:37.016193 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.016169    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:40:37.095827 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.095603    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-token\" (UniqueName: \"kubernetes.io/projected/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-token\") pod \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") "
Oct 30 04:40:37.095827 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.095700    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-data\") pod \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") "
Oct 30 04:40:37.095827 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.095780    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-podinfo\") pod \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") "
Oct 30 04:40:37.096120 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.095844    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-workload-socket\") pod \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") "
Oct 30 04:40:37.096120 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.095871    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-workload-certs\") pod \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") "
Oct 30 04:40:37.096120 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.095908    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istiod-ca-cert\") pod \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") "
Oct 30 04:40:37.096120 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.095951    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-credential-socket\") pod \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") "
Oct 30 04:40:37.096120 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.096010    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-kube-api-access-6zxgf\" (UniqueName: \"kubernetes.io/projected/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-kube-api-access-6zxgf\") pod \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") "
Oct 30 04:40:37.096421 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.096131    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-envoy\") pod \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\" (UID: \"b3e828ca-ccfa-471b-a2ee-4799f2edb09c\") "
Oct 30 04:40:37.096421 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.096384    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-data" pod "b3e828ca-ccfa-471b-a2ee-4799f2edb09c" (UID: "b3e828ca-ccfa-471b-a2ee-4799f2edb09c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:40:37.096789 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.096507    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-credential-socket" pod "b3e828ca-ccfa-471b-a2ee-4799f2edb09c" (UID: "b3e828ca-ccfa-471b-a2ee-4799f2edb09c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:40:37.096789 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.096567    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-workload-certs" pod "b3e828ca-ccfa-471b-a2ee-4799f2edb09c" (UID: "b3e828ca-ccfa-471b-a2ee-4799f2edb09c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:40:37.096789 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.096579    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istiod-ca-cert" pod "b3e828ca-ccfa-471b-a2ee-4799f2edb09c" (UID: "b3e828ca-ccfa-471b-a2ee-4799f2edb09c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:40:37.096789 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.096746    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-workload-socket" pod "b3e828ca-ccfa-471b-a2ee-4799f2edb09c" (UID: "b3e828ca-ccfa-471b-a2ee-4799f2edb09c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:40:37.097551 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.097462    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:37.097551 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.097487    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:37.097551 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.097502    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:37.097551 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.097516    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:37.097551 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.097528    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:37.098648 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.098609    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-podinfo" pod "b3e828ca-ccfa-471b-a2ee-4799f2edb09c" (UID: "b3e828ca-ccfa-471b-a2ee-4799f2edb09c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:40:37.098648 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.098627    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-token" pod "b3e828ca-ccfa-471b-a2ee-4799f2edb09c" (UID: "b3e828ca-ccfa-471b-a2ee-4799f2edb09c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:40:37.099556 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.099439    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-envoy" pod "b3e828ca-ccfa-471b-a2ee-4799f2edb09c" (UID: "b3e828ca-ccfa-471b-a2ee-4799f2edb09c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:40:37.099556 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.099523    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-kube-api-access-6zxgf" pod "b3e828ca-ccfa-471b-a2ee-4799f2edb09c" (UID: "b3e828ca-ccfa-471b-a2ee-4799f2edb09c"). InnerVolumeSpecName "kube-api-access-6zxgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:40:37.199094 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.199059    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:37.199094 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.199088    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zxgf\" (UniqueName: \"kubernetes.io/projected/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-kube-api-access-6zxgf\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:37.199094 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.199099    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:37.199336 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.199108    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3e828ca-ccfa-471b-a2ee-4799f2edb09c-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:40:37.239490 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.239457    4874 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 30 04:40:37.239847 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.239531    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p" podUID="c89aeca9-c4fc-4f9c-ba74-033b72349ddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 30 04:40:37.757271 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.757216    4874 generic.go:356] "Generic (PLEG): container finished" podID="b3e828ca-ccfa-471b-a2ee-4799f2edb09c" containerID="a84cc1a728f76def95c124aedbf8f320cba0899d25f57e7adc8129ea27fdaa94" exitCode=0
Oct 30 04:40:37.757271 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.757266    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq" event={"ID":"b3e828ca-ccfa-471b-a2ee-4799f2edb09c","Type":"ContainerDied","Data":"a84cc1a728f76def95c124aedbf8f320cba0899d25f57e7adc8129ea27fdaa94"}
Oct 30 04:40:37.757478 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.757287    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"
Oct 30 04:40:37.757478 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.757317    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq" event={"ID":"b3e828ca-ccfa-471b-a2ee-4799f2edb09c","Type":"ContainerDied","Data":"c4ff8700b96d3a1734f3020215170e55811f90aded23231388653d79f3a33b84"}
Oct 30 04:40:37.757478 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.757339    4874 scope.go:122] "RemoveContainer" containerID="a84cc1a728f76def95c124aedbf8f320cba0899d25f57e7adc8129ea27fdaa94"
Oct 30 04:40:37.792327 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.792302    4874 scope.go:122] "RemoveContainer" containerID="e815dc04d4160bc30860e7e30ee770dc0b935e3de0039ed90e429ced7e352d26"
Oct 30 04:40:37.801096 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.801078    4874 scope.go:122] "RemoveContainer" containerID="a84cc1a728f76def95c124aedbf8f320cba0899d25f57e7adc8129ea27fdaa94"
Oct 30 04:40:37.801382 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:40:37.801364    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a84cc1a728f76def95c124aedbf8f320cba0899d25f57e7adc8129ea27fdaa94\": container with ID starting with a84cc1a728f76def95c124aedbf8f320cba0899d25f57e7adc8129ea27fdaa94 not found: ID does not exist" containerID="a84cc1a728f76def95c124aedbf8f320cba0899d25f57e7adc8129ea27fdaa94"
Oct 30 04:40:37.801434 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.801391    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a84cc1a728f76def95c124aedbf8f320cba0899d25f57e7adc8129ea27fdaa94"} err="failed to get container status \"a84cc1a728f76def95c124aedbf8f320cba0899d25f57e7adc8129ea27fdaa94\": rpc error: code = NotFound desc = could not find container \"a84cc1a728f76def95c124aedbf8f320cba0899d25f57e7adc8129ea27fdaa94\": container with ID starting with a84cc1a728f76def95c124aedbf8f320cba0899d25f57e7adc8129ea27fdaa94 not found: ID does not exist"
Oct 30 04:40:37.801434 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.801408    4874 scope.go:122] "RemoveContainer" containerID="e815dc04d4160bc30860e7e30ee770dc0b935e3de0039ed90e429ced7e352d26"
Oct 30 04:40:37.801679 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:40:37.801659    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e815dc04d4160bc30860e7e30ee770dc0b935e3de0039ed90e429ced7e352d26\": container with ID starting with e815dc04d4160bc30860e7e30ee770dc0b935e3de0039ed90e429ced7e352d26 not found: ID does not exist" containerID="e815dc04d4160bc30860e7e30ee770dc0b935e3de0039ed90e429ced7e352d26"
Oct 30 04:40:37.801722 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.801685    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e815dc04d4160bc30860e7e30ee770dc0b935e3de0039ed90e429ced7e352d26"} err="failed to get container status \"e815dc04d4160bc30860e7e30ee770dc0b935e3de0039ed90e429ced7e352d26\": rpc error: code = NotFound desc = could not find container \"e815dc04d4160bc30860e7e30ee770dc0b935e3de0039ed90e429ced7e352d26\": container with ID starting with e815dc04d4160bc30860e7e30ee770dc0b935e3de0039ed90e429ced7e352d26 not found: ID does not exist"
Oct 30 04:40:37.829770 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.829745    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"]
Oct 30 04:40:37.834608 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:37.834589    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-2p6tq"]
Oct 30 04:40:38.239446 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:38.239411    4874 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 30 04:40:38.239589 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:38.239492    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p" podUID="c89aeca9-c4fc-4f9c-ba74-033b72349ddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 30 04:40:38.730530 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:38.730501    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3e828ca-ccfa-471b-a2ee-4799f2edb09c" path="/var/lib/kubelet/pods/b3e828ca-ccfa-471b-a2ee-4799f2edb09c/volumes"
Oct 30 04:40:38.736127 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:38.736104    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s"
Oct 30 04:40:39.238404 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:39.238370    4874 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 30 04:40:39.238567 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:39.238430    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p" podUID="c89aeca9-c4fc-4f9c-ba74-033b72349ddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 30 04:40:40.238745 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:40.238712    4874 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 30 04:40:40.239169 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:40.238760    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p" podUID="c89aeca9-c4fc-4f9c-ba74-033b72349ddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 30 04:40:41.242669 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:41.242632    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:41.243159 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:41.242944    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:41.243982 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:41.243957    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 04:40:42.729746 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:42.729714    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj"
Oct 30 04:40:47.182440 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:47.182407    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27"]
Oct 30 04:40:47.189192 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:47.189164    4874 cpu_manager.go:414] "containerMap: removing container" podUID="b3e828ca-ccfa-471b-a2ee-4799f2edb09c" containerName="istio-proxy"
Oct 30 04:40:47.189339 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:47.189285    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3e828ca-ccfa-471b-a2ee-4799f2edb09c" containerName="istio-proxy"
Oct 30 04:40:47.189402 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:47.189345    4874 cpu_manager.go:414] "containerMap: removing container" podUID="b3e828ca-ccfa-471b-a2ee-4799f2edb09c" containerName="istio-proxy"
Oct 30 04:40:47.189402 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:47.189353    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3e828ca-ccfa-471b-a2ee-4799f2edb09c" containerName="istio-proxy"
Oct 30 04:40:47.189512 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:47.189483    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3e828ca-ccfa-471b-a2ee-4799f2edb09c" containerName="istio-proxy"
Oct 30 04:40:47.193194 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:47.193173    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27"
Oct 30 04:40:47.207290 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:47.207268    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27"]
Oct 30 04:40:47.221794 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:47.216300    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-247158-serving-cert\""
Oct 30 04:40:47.231519 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:47.231452    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14f989b1-ee76-4cef-950c-ed97b8a5b19b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27\" (UID: \"14f989b1-ee76-4cef-950c-ed97b8a5b19b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27"
Oct 30 04:40:47.231519 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:47.231495    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14f989b1-ee76-4cef-950c-ed97b8a5b19b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27\" (UID: \"14f989b1-ee76-4cef-950c-ed97b8a5b19b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27"
Oct 30 04:40:47.231689 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:47.231646    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14f989b1-ee76-4cef-950c-ed97b8a5b19b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27\" (UID: \"14f989b1-ee76-4cef-950c-ed97b8a5b19b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27"
Oct 30 04:40:47.231753 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:47.231699    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4bl5\" (UniqueName: \"kubernetes.io/projected/14f989b1-ee76-4cef-950c-ed97b8a5b19b-kube-api-access-m4bl5\") pod \"maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27\" (UID: \"14f989b1-ee76-4cef-950c-ed97b8a5b19b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27"
Oct 30 04:40:47.332919 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:47.332887    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14f989b1-ee76-4cef-950c-ed97b8a5b19b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27\" (UID: \"14f989b1-ee76-4cef-950c-ed97b8a5b19b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27"
Oct 30 04:40:47.333090 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:47.332929    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14f989b1-ee76-4cef-950c-ed97b8a5b19b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27\" (UID: \"14f989b1-ee76-4cef-950c-ed97b8a5b19b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27"
Oct 30 04:40:47.333090 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:47.332981    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14f989b1-ee76-4cef-950c-ed97b8a5b19b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27\" (UID: \"14f989b1-ee76-4cef-950c-ed97b8a5b19b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27"
Oct 30 04:40:47.333090 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:47.333005    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4bl5\" (UniqueName: \"kubernetes.io/projected/14f989b1-ee76-4cef-950c-ed97b8a5b19b-kube-api-access-m4bl5\") pod \"maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27\" (UID: \"14f989b1-ee76-4cef-950c-ed97b8a5b19b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27"
Oct 30 04:40:47.333549 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:47.333527    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14f989b1-ee76-4cef-950c-ed97b8a5b19b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27\" (UID: \"14f989b1-ee76-4cef-950c-ed97b8a5b19b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27"
Oct 30 04:40:47.335489 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:47.335465    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14f989b1-ee76-4cef-950c-ed97b8a5b19b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27\" (UID: \"14f989b1-ee76-4cef-950c-ed97b8a5b19b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27"
Oct 30 04:40:47.335595 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:47.335515    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14f989b1-ee76-4cef-950c-ed97b8a5b19b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27\" (UID: \"14f989b1-ee76-4cef-950c-ed97b8a5b19b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27"
Oct 30 04:40:47.355358 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:47.355331    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4bl5\" (UniqueName: \"kubernetes.io/projected/14f989b1-ee76-4cef-950c-ed97b8a5b19b-kube-api-access-m4bl5\") pod \"maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27\" (UID: \"14f989b1-ee76-4cef-950c-ed97b8a5b19b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27"
Oct 30 04:40:47.505807 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:47.505777    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27"
Oct 30 04:40:47.812462 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:47.812386    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27"]
Oct 30 04:40:47.814860 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:40:47.814833    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f989b1_ee76_4cef_950c_ed97b8a5b19b.slice/crio-b505d9c39e73d94bf1d9054420b1757b554e1b086907723739a80eb4abdb343e WatchSource:0}: Error finding container b505d9c39e73d94bf1d9054420b1757b554e1b086907723739a80eb4abdb343e: Status 404 returned error can't find the container with id b505d9c39e73d94bf1d9054420b1757b554e1b086907723739a80eb4abdb343e
Oct 30 04:40:47.838551 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:47.838527    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27" event={"ID":"14f989b1-ee76-4cef-950c-ed97b8a5b19b","Type":"ContainerStarted","Data":"b505d9c39e73d94bf1d9054420b1757b554e1b086907723739a80eb4abdb343e"}
Oct 30 04:40:48.854525 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:48.854486    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27" event={"ID":"14f989b1-ee76-4cef-950c-ed97b8a5b19b","Type":"ContainerStarted","Data":"0a136b637e470720fad8a50c08f21e65d24e76b05fe51f3753dc09eb46939691"}
Oct 30 04:40:48.854886 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:48.854585    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27"
Oct 30 04:40:48.897431 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:48.897388    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27" podStartSLOduration=1.557470627 podStartE2EDuration="1.897375991s" totalImagesPullingTime="339.905364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:40:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:40:48.895807513 +0000 UTC m=+2568.308203087" watchObservedRunningTime="2026-10-30 04:40:48.897375991 +0000 UTC m=+2568.309771605"
Oct 30 04:40:50.918365 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:50.918331    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"]
Oct 30 04:40:50.919212 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:50.919191    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3e828ca-ccfa-471b-a2ee-4799f2edb09c" containerName="istio-proxy"
Oct 30 04:40:50.921936 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:50.921920    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:50.936884 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:50.936862    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-4c2415-openshift-default-dockercfg-rrdsn\""
Oct 30 04:40:50.950970 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:50.950946    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"]
Oct 30 04:40:50.981682 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:50.981653    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-data\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:50.981799 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:50.981705    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-token\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:50.981959 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:50.981932    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnqct\" (UniqueName: \"kubernetes.io/projected/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-kube-api-access-xnqct\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:50.982061 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:50.982043    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-workload-socket\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:50.982105 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:50.982090    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-credential-socket\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:50.982177 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:50.982155    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-envoy\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:50.982250 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:50.982220    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-workload-certs\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:50.982314 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:50.982295    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istiod-ca-cert\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:50.982348 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:50.982337    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-podinfo\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:51.088806 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.083898    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-token\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:51.088806 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.084062    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnqct\" (UniqueName: \"kubernetes.io/projected/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-kube-api-access-xnqct\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:51.088806 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.084135    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-workload-socket\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:51.088806 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.084177    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-credential-socket\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:51.088806 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.084204    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-envoy\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:51.088806 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.084289    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-workload-certs\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:51.088806 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.084314    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istiod-ca-cert\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:51.088806 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.084358    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-podinfo\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:51.088806 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.084409    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-data\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:51.088806 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.084839    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-data\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:51.088806 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.086221    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-workload-socket\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:51.088806 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.086512    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-credential-socket\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:51.088806 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.087982    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istiod-ca-cert\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:51.088806 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.088372    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-workload-certs\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:51.103259 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.095332    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-envoy\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:51.104068 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.104042    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-podinfo\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:51.114179 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.114151    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnqct\" (UniqueName: \"kubernetes.io/projected/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-kube-api-access-xnqct\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:51.115723 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.115693    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-token\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:51.236251 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.236206    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:51.562325 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.562163    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"]
Oct 30 04:40:51.564617 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:40:51.564587    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac7a2fc_ad35_41f6_babc_297ab9f7a97e.slice/crio-a9a41c74e6c75a1b4cb83230bd3f09e5cd0eba4c6ca507ca8c89608affabb287 WatchSource:0}: Error finding container a9a41c74e6c75a1b4cb83230bd3f09e5cd0eba4c6ca507ca8c89608affabb287: Status 404 returned error can't find the container with id a9a41c74e6c75a1b4cb83230bd3f09e5cd0eba4c6ca507ca8c89608affabb287
Oct 30 04:40:51.567086 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.567056    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:40:51.567177 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.567114    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:40:51.567177 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.567163    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:40:51.825861 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.825779    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:40:51.874426 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.874377    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss" event={"ID":"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e","Type":"ContainerStarted","Data":"36e5bfe5090d29adf31c44d42828df62d4b0565b72a3a848897d9f13298fb244"}
Oct 30 04:40:51.874596 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.874452    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss" event={"ID":"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e","Type":"ContainerStarted","Data":"a9a41c74e6c75a1b4cb83230bd3f09e5cd0eba4c6ca507ca8c89608affabb287"}
Oct 30 04:40:51.919151 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:51.919088    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss" podStartSLOduration=1.919067637 podStartE2EDuration="1.919067637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:40:51.917997493 +0000 UTC m=+2571.330393086" watchObservedRunningTime="2026-10-30 04:40:51.919067637 +0000 UTC m=+2571.331463213"
Oct 30 04:40:52.239367 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:52.238888    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:52.264741 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:52.264595    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:52.895988 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:52.895950    4874 generic.go:356] "Generic (PLEG): container finished" podID="d5587cde-1301-458b-80c5-5a83682b3e21" containerID="94a89367cef3a0bb978ace6f7734f152e213169061d6bf73441c286f51180167" exitCode=0
Oct 30 04:40:52.896202 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:52.896018    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h" event={"ID":"d5587cde-1301-458b-80c5-5a83682b3e21","Type":"ContainerDied","Data":"94a89367cef3a0bb978ace6f7734f152e213169061d6bf73441c286f51180167"}
Oct 30 04:40:52.896527 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:52.896502    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:52.896800 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:52.896775    4874 scope.go:122] "RemoveContainer" containerID="94a89367cef3a0bb978ace6f7734f152e213169061d6bf73441c286f51180167"
Oct 30 04:40:52.898153 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:52.898133    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:40:52.898747 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:52.898710    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:40:52.898838 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:52.898797    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:40:52.898900 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:52.898846    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:40:53.903530 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:53.903496    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h" event={"ID":"d5587cde-1301-458b-80c5-5a83682b3e21","Type":"ContainerStarted","Data":"29159fcdcf1a27941446d90960fd804b6bac1b4db837360519aa6865bdce46b2"}
Oct 30 04:40:54.416944 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:54.416908    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:54.418169 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:54.418141    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 04:40:54.418309 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:54.418196    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h" podUID="d5587cde-1301-458b-80c5-5a83682b3e21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 04:40:54.888416 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:54.888384    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27"
Oct 30 04:40:55.417764 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:55.417729    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 04:40:55.418288 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:55.417787    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h" podUID="d5587cde-1301-458b-80c5-5a83682b3e21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 04:40:55.615588 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:40:55.615552    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:40:56.417980 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:56.417948    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 04:40:56.418451 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:56.417999    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h" podUID="d5587cde-1301-458b-80c5-5a83682b3e21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 04:40:57.422688 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:57.422645    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 04:40:57.423359 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:57.423327    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h" podUID="d5587cde-1301-458b-80c5-5a83682b3e21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 04:40:57.544287 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:57.544256    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:58.418089 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:58.418044    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 04:40:58.418290 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:58.418106    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h" podUID="d5587cde-1301-458b-80c5-5a83682b3e21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 04:40:59.421593 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:59.421563    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:40:59.422650 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:40:59.422627    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 04:41:12.539985 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:12.539948    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:41:24.126536 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:24.126498    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"]
Oct 30 04:41:24.127053 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:24.126794    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89" podUID="7dabf1f5-a781-45e8-a9f2-622109ab647b" containerName="istio-proxy" containerID="cri-o://40b8ecfa1e6591f9a24acede1c7983e8a2d2edd2f0b024bbcf5c01e5f093e49f" gracePeriod=30
Oct 30 04:41:25.608627 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:41:25.608593    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:41:25.740697 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:25.740661    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4"]
Oct 30 04:41:25.745315 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:25.745291    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4"
Oct 30 04:41:25.772384 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:25.772355    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-20a650-dockercfg-87gmb\""
Oct 30 04:41:25.774745 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:25.774720    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4"]
Oct 30 04:41:25.800817 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:25.800763    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-20a650\""
Oct 30 04:41:25.876048 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:25.875951    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37a947d-42b4-4b02-a8bb-17db1cda4b85-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4\" (UID: \"b37a947d-42b4-4b02-a8bb-17db1cda4b85\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4"
Oct 30 04:41:25.876533 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:25.876494    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzdrf\" (UniqueName: \"kubernetes.io/projected/b37a947d-42b4-4b02-a8bb-17db1cda4b85-kube-api-access-wzdrf\") pod \"payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4\" (UID: \"b37a947d-42b4-4b02-a8bb-17db1cda4b85\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4"
Oct 30 04:41:25.977996 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:25.977967    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37a947d-42b4-4b02-a8bb-17db1cda4b85-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4\" (UID: \"b37a947d-42b4-4b02-a8bb-17db1cda4b85\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4"
Oct 30 04:41:25.978195 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:25.978095    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzdrf\" (UniqueName: \"kubernetes.io/projected/b37a947d-42b4-4b02-a8bb-17db1cda4b85-kube-api-access-wzdrf\") pod \"payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4\" (UID: \"b37a947d-42b4-4b02-a8bb-17db1cda4b85\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4"
Oct 30 04:41:25.978756 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:25.978729    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37a947d-42b4-4b02-a8bb-17db1cda4b85-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4\" (UID: \"b37a947d-42b4-4b02-a8bb-17db1cda4b85\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4"
Oct 30 04:41:26.000627 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:26.000597    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzdrf\" (UniqueName: \"kubernetes.io/projected/b37a947d-42b4-4b02-a8bb-17db1cda4b85-kube-api-access-wzdrf\") pod \"payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4\" (UID: \"b37a947d-42b4-4b02-a8bb-17db1cda4b85\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4"
Oct 30 04:41:26.057473 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:26.057445    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4"
Oct 30 04:41:26.357987 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:26.357964    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4"]
Oct 30 04:41:26.360767 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:41:26.360740    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37a947d_42b4_4b02_a8bb_17db1cda4b85.slice/crio-99a5ecdedc03822c06b9b5b6237dc55a28e4ec06c88ad09fba83e94163149fd9 WatchSource:0}: Error finding container 99a5ecdedc03822c06b9b5b6237dc55a28e4ec06c88ad09fba83e94163149fd9: Status 404 returned error can't find the container with id 99a5ecdedc03822c06b9b5b6237dc55a28e4ec06c88ad09fba83e94163149fd9
Oct 30 04:41:27.110536 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:27.110493    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4" event={"ID":"b37a947d-42b4-4b02-a8bb-17db1cda4b85","Type":"ContainerStarted","Data":"b0aab1513a0adc2e82a7816c6ba52c7fe5e9d2f67242b46685a7678426c655ad"}
Oct 30 04:41:27.110940 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:27.110554    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4" event={"ID":"b37a947d-42b4-4b02-a8bb-17db1cda4b85","Type":"ContainerStarted","Data":"99a5ecdedc03822c06b9b5b6237dc55a28e4ec06c88ad09fba83e94163149fd9"}
Oct 30 04:41:27.110940 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:27.110599    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4"
Oct 30 04:41:27.153347 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:27.153289    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4" podStartSLOduration=2.153274626 podStartE2EDuration="2.153274626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:41:27.150822748 +0000 UTC m=+2606.563218324" watchObservedRunningTime="2026-10-30 04:41:27.153274626 +0000 UTC m=+2606.565670198"
Oct 30 04:41:29.423627 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.423601    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:41:29.555980 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.555948    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-credential-socket\") pod \"7dabf1f5-a781-45e8-a9f2-622109ab647b\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") "
Oct 30 04:41:29.556165 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.556005    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-workload-socket\") pod \"7dabf1f5-a781-45e8-a9f2-622109ab647b\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") "
Oct 30 04:41:29.556165 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.556026    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-envoy\") pod \"7dabf1f5-a781-45e8-a9f2-622109ab647b\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") "
Oct 30 04:41:29.556165 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.556063    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-workload-certs\") pod \"7dabf1f5-a781-45e8-a9f2-622109ab647b\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") "
Oct 30 04:41:29.556165 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.556085    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-data\") pod \"7dabf1f5-a781-45e8-a9f2-622109ab647b\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") "
Oct 30 04:41:29.556420 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.556195    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-token\" (UniqueName: \"kubernetes.io/projected/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-token\") pod \"7dabf1f5-a781-45e8-a9f2-622109ab647b\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") "
Oct 30 04:41:29.556420 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.556263    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dabf1f5-a781-45e8-a9f2-622109ab647b-kube-api-access-xnsb2\" (UniqueName: \"kubernetes.io/projected/7dabf1f5-a781-45e8-a9f2-622109ab647b-kube-api-access-xnsb2\") pod \"7dabf1f5-a781-45e8-a9f2-622109ab647b\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") "
Oct 30 04:41:29.556420 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.556291    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-workload-socket" pod "7dabf1f5-a781-45e8-a9f2-622109ab647b" (UID: "7dabf1f5-a781-45e8-a9f2-622109ab647b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:41:29.556420 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.556313    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-podinfo\") pod \"7dabf1f5-a781-45e8-a9f2-622109ab647b\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") "
Oct 30 04:41:29.556420 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.556293    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-credential-socket" pod "7dabf1f5-a781-45e8-a9f2-622109ab647b" (UID: "7dabf1f5-a781-45e8-a9f2-622109ab647b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:41:29.556420 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.556381    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dabf1f5-a781-45e8-a9f2-622109ab647b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dabf1f5-a781-45e8-a9f2-622109ab647b-istiod-ca-cert\") pod \"7dabf1f5-a781-45e8-a9f2-622109ab647b\" (UID: \"7dabf1f5-a781-45e8-a9f2-622109ab647b\") "
Oct 30 04:41:29.556420 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.556394    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-workload-certs" pod "7dabf1f5-a781-45e8-a9f2-622109ab647b" (UID: "7dabf1f5-a781-45e8-a9f2-622109ab647b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:41:29.556420 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.556403    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-data" pod "7dabf1f5-a781-45e8-a9f2-622109ab647b" (UID: "7dabf1f5-a781-45e8-a9f2-622109ab647b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:41:29.556825 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.556723    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dabf1f5-a781-45e8-a9f2-622109ab647b-istiod-ca-cert" pod "7dabf1f5-a781-45e8-a9f2-622109ab647b" (UID: "7dabf1f5-a781-45e8-a9f2-622109ab647b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:41:29.556947 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.556928    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dabf1f5-a781-45e8-a9f2-622109ab647b-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:41:29.557031 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.556951    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:41:29.557031 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.556966    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:41:29.557031 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.556979    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:41:29.557031 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.556992    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:41:29.558502 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.558469    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-envoy" pod "7dabf1f5-a781-45e8-a9f2-622109ab647b" (UID: "7dabf1f5-a781-45e8-a9f2-622109ab647b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:41:29.558606 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.558540    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-token" pod "7dabf1f5-a781-45e8-a9f2-622109ab647b" (UID: "7dabf1f5-a781-45e8-a9f2-622109ab647b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:41:29.558606 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.558556    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-podinfo" pod "7dabf1f5-a781-45e8-a9f2-622109ab647b" (UID: "7dabf1f5-a781-45e8-a9f2-622109ab647b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:41:29.558606 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.558565    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dabf1f5-a781-45e8-a9f2-622109ab647b-kube-api-access-xnsb2" pod "7dabf1f5-a781-45e8-a9f2-622109ab647b" (UID: "7dabf1f5-a781-45e8-a9f2-622109ab647b"). InnerVolumeSpecName "kube-api-access-xnsb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:41:29.658649 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.658571    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:41:29.658649 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.658599    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:41:29.658649 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.658608    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnsb2\" (UniqueName: \"kubernetes.io/projected/7dabf1f5-a781-45e8-a9f2-622109ab647b-kube-api-access-xnsb2\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:41:29.658649 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:29.658619    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dabf1f5-a781-45e8-a9f2-622109ab647b-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:41:30.129290 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:30.129254    4874 generic.go:356] "Generic (PLEG): container finished" podID="7dabf1f5-a781-45e8-a9f2-622109ab647b" containerID="40b8ecfa1e6591f9a24acede1c7983e8a2d2edd2f0b024bbcf5c01e5f093e49f" exitCode=0
Oct 30 04:41:30.129290 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:30.129293    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89" event={"ID":"7dabf1f5-a781-45e8-a9f2-622109ab647b","Type":"ContainerDied","Data":"40b8ecfa1e6591f9a24acede1c7983e8a2d2edd2f0b024bbcf5c01e5f093e49f"}
Oct 30 04:41:30.129524 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:30.129307    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"
Oct 30 04:41:30.129524 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:30.129336    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89" event={"ID":"7dabf1f5-a781-45e8-a9f2-622109ab647b","Type":"ContainerDied","Data":"e06b9ae26e61c517f3fcb06971df6208afe882a068f8e7e50e67782c5a78617b"}
Oct 30 04:41:30.129524 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:30.129357    4874 scope.go:122] "RemoveContainer" containerID="40b8ecfa1e6591f9a24acede1c7983e8a2d2edd2f0b024bbcf5c01e5f093e49f"
Oct 30 04:41:30.139633 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:30.139616    4874 scope.go:122] "RemoveContainer" containerID="40b8ecfa1e6591f9a24acede1c7983e8a2d2edd2f0b024bbcf5c01e5f093e49f"
Oct 30 04:41:30.139909 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:41:30.139891    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40b8ecfa1e6591f9a24acede1c7983e8a2d2edd2f0b024bbcf5c01e5f093e49f\": container with ID starting with 40b8ecfa1e6591f9a24acede1c7983e8a2d2edd2f0b024bbcf5c01e5f093e49f not found: ID does not exist" containerID="40b8ecfa1e6591f9a24acede1c7983e8a2d2edd2f0b024bbcf5c01e5f093e49f"
Oct 30 04:41:30.139973 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:30.139919    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40b8ecfa1e6591f9a24acede1c7983e8a2d2edd2f0b024bbcf5c01e5f093e49f"} err="failed to get container status \"40b8ecfa1e6591f9a24acede1c7983e8a2d2edd2f0b024bbcf5c01e5f093e49f\": rpc error: code = NotFound desc = could not find container \"40b8ecfa1e6591f9a24acede1c7983e8a2d2edd2f0b024bbcf5c01e5f093e49f\": container with ID starting with 40b8ecfa1e6591f9a24acede1c7983e8a2d2edd2f0b024bbcf5c01e5f093e49f not found: ID does not exist"
Oct 30 04:41:30.212159 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:30.212129    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"]
Oct 30 04:41:30.214757 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:30.214726    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-6cb89"]
Oct 30 04:41:30.728750 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:30.728716    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dabf1f5-a781-45e8-a9f2-622109ab647b" path="/var/lib/kubelet/pods/7dabf1f5-a781-45e8-a9f2-622109ab647b/volumes"
Oct 30 04:41:35.178949 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.178905    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"]
Oct 30 04:41:35.179445 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.179434    4874 cpu_manager.go:414] "containerMap: removing container" podUID="7dabf1f5-a781-45e8-a9f2-622109ab647b" containerName="istio-proxy"
Oct 30 04:41:35.179528 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.179446    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dabf1f5-a781-45e8-a9f2-622109ab647b" containerName="istio-proxy"
Oct 30 04:41:35.179587 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.179548    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dabf1f5-a781-45e8-a9f2-622109ab647b" containerName="istio-proxy"
Oct 30 04:41:35.184343 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.184322    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:41:35.190461 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.190438    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"]
Oct 30 04:41:35.198880 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.198859    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-495ebe\"/\"default-dockercfg-lclwq\""
Oct 30 04:41:35.199697 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.199084    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-495ebe\"/\"auth-test-model-495ebe-kserve-self-signed-certs\""
Oct 30 04:41:35.209943 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.209922    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-495ebe\"/\"kube-root-ca.crt\""
Oct 30 04:41:35.210055 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.210016    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-495ebe\"/\"openshift-service-ca.crt\""
Oct 30 04:41:35.245609 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.245586    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-home\") pod \"auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz\" (UID: \"5e850259-0899-4e15-93eb-d3f861d16b81\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:41:35.245749 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.245622    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-tmp-dir\") pod \"auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz\" (UID: \"5e850259-0899-4e15-93eb-d3f861d16b81\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:41:35.245821 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.245785    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e850259-0899-4e15-93eb-d3f861d16b81-tls-certs\") pod \"auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz\" (UID: \"5e850259-0899-4e15-93eb-d3f861d16b81\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:41:35.245896 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.245834    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-model-cache\") pod \"auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz\" (UID: \"5e850259-0899-4e15-93eb-d3f861d16b81\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:41:35.245949 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.245880    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64sqb\" (UniqueName: \"kubernetes.io/projected/5e850259-0899-4e15-93eb-d3f861d16b81-kube-api-access-64sqb\") pod \"auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz\" (UID: \"5e850259-0899-4e15-93eb-d3f861d16b81\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:41:35.246074 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.246048    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-dshm\") pod \"auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz\" (UID: \"5e850259-0899-4e15-93eb-d3f861d16b81\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:41:35.346711 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.346679    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e850259-0899-4e15-93eb-d3f861d16b81-tls-certs\") pod \"auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz\" (UID: \"5e850259-0899-4e15-93eb-d3f861d16b81\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:41:35.346857 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.346732    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-model-cache\") pod \"auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz\" (UID: \"5e850259-0899-4e15-93eb-d3f861d16b81\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:41:35.346857 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.346768    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64sqb\" (UniqueName: \"kubernetes.io/projected/5e850259-0899-4e15-93eb-d3f861d16b81-kube-api-access-64sqb\") pod \"auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz\" (UID: \"5e850259-0899-4e15-93eb-d3f861d16b81\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:41:35.346857 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.346821    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-dshm\") pod \"auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz\" (UID: \"5e850259-0899-4e15-93eb-d3f861d16b81\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:41:35.347026 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.346860    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-home\") pod \"auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz\" (UID: \"5e850259-0899-4e15-93eb-d3f861d16b81\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:41:35.347026 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.346887    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-tmp-dir\") pod \"auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz\" (UID: \"5e850259-0899-4e15-93eb-d3f861d16b81\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:41:35.347230 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.347202    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-model-cache\") pod \"auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz\" (UID: \"5e850259-0899-4e15-93eb-d3f861d16b81\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:41:35.347341 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.347227    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-tmp-dir\") pod \"auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz\" (UID: \"5e850259-0899-4e15-93eb-d3f861d16b81\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:41:35.347408 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.347338    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-home\") pod \"auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz\" (UID: \"5e850259-0899-4e15-93eb-d3f861d16b81\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:41:35.349089 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.349058    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-dshm\") pod \"auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz\" (UID: \"5e850259-0899-4e15-93eb-d3f861d16b81\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:41:35.349309 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.349289    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e850259-0899-4e15-93eb-d3f861d16b81-tls-certs\") pod \"auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz\" (UID: \"5e850259-0899-4e15-93eb-d3f861d16b81\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:41:35.368768 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.368749    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64sqb\" (UniqueName: \"kubernetes.io/projected/5e850259-0899-4e15-93eb-d3f861d16b81-kube-api-access-64sqb\") pod \"auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz\" (UID: \"5e850259-0899-4e15-93eb-d3f861d16b81\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:41:35.505779 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.505742    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:41:35.817028 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:35.817001    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"]
Oct 30 04:41:35.818931 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:41:35.818906    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e850259_0899_4e15_93eb_d3f861d16b81.slice/crio-af23a1373478865893fc05ca01c48b8dd289fb00a67e426330602cd16944129e WatchSource:0}: Error finding container af23a1373478865893fc05ca01c48b8dd289fb00a67e426330602cd16944129e: Status 404 returned error can't find the container with id af23a1373478865893fc05ca01c48b8dd289fb00a67e426330602cd16944129e
Oct 30 04:41:36.175581 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:36.175499    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz" event={"ID":"5e850259-0899-4e15-93eb-d3f861d16b81","Type":"ContainerStarted","Data":"dd9bf5f144d2f42e01cc9a3bd5e7baf05b5764e27e3688cf9263bc0d217faedc"}
Oct 30 04:41:36.175581 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:36.175544    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz" event={"ID":"5e850259-0899-4e15-93eb-d3f861d16b81","Type":"ContainerStarted","Data":"af23a1373478865893fc05ca01c48b8dd289fb00a67e426330602cd16944129e"}
Oct 30 04:41:36.215430 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:36.215382    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz" podStartSLOduration=1.215367984 podStartE2EDuration="1.215367984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:41:36.214597991 +0000 UTC m=+2615.626993563" watchObservedRunningTime="2026-10-30 04:41:36.215367984 +0000 UTC m=+2615.627763556"
Oct 30 04:41:36.506617 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:36.506585    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:41:37.182816 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:37.182782    4874 generic.go:356] "Generic (PLEG): container finished" podID="6ac7a2fc-ad35-41f6-babc-297ab9f7a97e" containerID="36e5bfe5090d29adf31c44d42828df62d4b0565b72a3a848897d9f13298fb244" exitCode=0
Oct 30 04:41:37.182990 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:37.182857    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss" event={"ID":"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e","Type":"ContainerDied","Data":"36e5bfe5090d29adf31c44d42828df62d4b0565b72a3a848897d9f13298fb244"}
Oct 30 04:41:37.183424 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:37.183401    4874 scope.go:122] "RemoveContainer" containerID="36e5bfe5090d29adf31c44d42828df62d4b0565b72a3a848897d9f13298fb244"
Oct 30 04:41:37.184743 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:37.184716    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:41:37.184814 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:37.184780    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:41:37.184868 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:37.184811    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:41:37.897100 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:37.897060    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:41:38.119591 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:38.119562    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4"
Oct 30 04:41:38.203492 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:38.203401    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss" event={"ID":"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e","Type":"ContainerStarted","Data":"b4dd3c7c14e3397330f15c411b606d08d74b7110247abfd86d391d229d807f4d"}
Oct 30 04:41:38.237144 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:38.237116    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:41:38.238367 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:38.238339    4874 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 30 04:41:38.238498 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:38.238390    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss" podUID="6ac7a2fc-ad35-41f6-babc-297ab9f7a97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 30 04:41:39.237065 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:39.237031    4874 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 30 04:41:39.237467 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:39.237080    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss" podUID="6ac7a2fc-ad35-41f6-babc-297ab9f7a97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 30 04:41:40.236736 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:40.236707    4874 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 30 04:41:40.236902 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:40.236752    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss" podUID="6ac7a2fc-ad35-41f6-babc-297ab9f7a97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 30 04:41:41.237370 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:41.237335    4874 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 30 04:41:41.237758 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:41.237386    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss" podUID="6ac7a2fc-ad35-41f6-babc-297ab9f7a97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 30 04:41:42.236584 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:42.236550    4874 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 30 04:41:42.236770 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:42.236598    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss" podUID="6ac7a2fc-ad35-41f6-babc-297ab9f7a97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 30 04:41:43.483812 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:43.483773    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:41:43.484514 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:43.484487    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:41:43.486143 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:43.486121    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 04:41:45.505959 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:45.505922    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:41:45.518923 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:45.518896    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:41:45.921131 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:45.921051    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"]
Oct 30 04:41:45.924103 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:45.924079    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:45.944052 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:45.944024    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-bbce72-openshift-default-dockercfg-6k6w8\""
Oct 30 04:41:45.981448 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:45.981415    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-credential-socket\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:45.981605 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:45.981452    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx5kh\" (UniqueName: \"kubernetes.io/projected/fde6ef20-18b2-4615-952e-8de495f731fe-kube-api-access-rx5kh\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:45.981605 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:45.981470    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-workload-socket\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:45.981605 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:45.981567    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-workload-certs\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:45.981773 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:45.981697    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fde6ef20-18b2-4615-952e-8de495f731fe-istiod-ca-cert\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:45.981837 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:45.981786    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-istio-envoy\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:45.981899 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:45.981831    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fde6ef20-18b2-4615-952e-8de495f731fe-istio-podinfo\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:45.981899 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:45.981886    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-istio-data\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:45.981970 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:45.981938    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fde6ef20-18b2-4615-952e-8de495f731fe-istio-token\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:45.998404 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:45.998377    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"]
Oct 30 04:41:46.083956 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.083900    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-istio-envoy\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:46.084146 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.083964    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fde6ef20-18b2-4615-952e-8de495f731fe-istio-podinfo\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:46.084146 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.084021    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-istio-data\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:46.084146 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.084087    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fde6ef20-18b2-4615-952e-8de495f731fe-istio-token\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:46.084146 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.084126    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-credential-socket\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:46.084400 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.084156    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx5kh\" (UniqueName: \"kubernetes.io/projected/fde6ef20-18b2-4615-952e-8de495f731fe-kube-api-access-rx5kh\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:46.084400 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.084187    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-workload-socket\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:46.084400 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.084218    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-workload-certs\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:46.084555 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.084398    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fde6ef20-18b2-4615-952e-8de495f731fe-istiod-ca-cert\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:46.085255 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.085209    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fde6ef20-18b2-4615-952e-8de495f731fe-istiod-ca-cert\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:46.086038 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.086015    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-credential-socket\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:46.086598 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.086571    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-istio-data\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:46.099335 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.086683    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-workload-socket\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:46.099572 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.086903    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-workload-certs\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:46.108060 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.107872    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fde6ef20-18b2-4615-952e-8de495f731fe-istio-podinfo\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:46.115174 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.114949    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-istio-envoy\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:46.120187 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.120159    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fde6ef20-18b2-4615-952e-8de495f731fe-istio-token\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:46.121035 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.120946    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx5kh\" (UniqueName: \"kubernetes.io/projected/fde6ef20-18b2-4615-952e-8de495f731fe-kube-api-access-rx5kh\") pod \"e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") " pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:46.239845 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.238507    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:46.260531 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.260506    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:41:46.562684 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.562658    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"]
Oct 30 04:41:46.565155 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:41:46.565134    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde6ef20_18b2_4615_952e_8de495f731fe.slice/crio-dc9d3cd0bc646b6afad93648b72085a85c95b5c112f232dc1c344ab272da6d51 WatchSource:0}: Error finding container dc9d3cd0bc646b6afad93648b72085a85c95b5c112f232dc1c344ab272da6d51: Status 404 returned error can't find the container with id dc9d3cd0bc646b6afad93648b72085a85c95b5c112f232dc1c344ab272da6d51
Oct 30 04:41:46.567746 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.567714    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:41:46.567828 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.567779    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:41:46.567828 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.567810    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:41:46.764254 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.764197    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg"]
Oct 30 04:41:46.767415 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.767385    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg"
Oct 30 04:41:46.780944 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.780915    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-20a650-serving-cert\""
Oct 30 04:41:46.787997 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.787967    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg"]
Oct 30 04:41:46.893706 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.893613    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-openshift-service-ca\") pod \"maas-api-e2e-explicit-20a650-666dffddc9-928xg\" (UID: \"a21aa3c4-4210-4c04-be28-16e0b6ffa6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg"
Oct 30 04:41:46.893706 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.893661    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-metrics-certs\") pod \"maas-api-e2e-explicit-20a650-666dffddc9-928xg\" (UID: \"a21aa3c4-4210-4c04-be28-16e0b6ffa6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg"
Oct 30 04:41:46.893893 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.893775    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-maas-api-tls\") pod \"maas-api-e2e-explicit-20a650-666dffddc9-928xg\" (UID: \"a21aa3c4-4210-4c04-be28-16e0b6ffa6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg"
Oct 30 04:41:46.893893 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.893836    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpwvg\" (UniqueName: \"kubernetes.io/projected/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-kube-api-access-cpwvg\") pod \"maas-api-e2e-explicit-20a650-666dffddc9-928xg\" (UID: \"a21aa3c4-4210-4c04-be28-16e0b6ffa6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg"
Oct 30 04:41:46.999437 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.998454    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-openshift-service-ca\") pod \"maas-api-e2e-explicit-20a650-666dffddc9-928xg\" (UID: \"a21aa3c4-4210-4c04-be28-16e0b6ffa6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg"
Oct 30 04:41:46.999437 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.998509    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-metrics-certs\") pod \"maas-api-e2e-explicit-20a650-666dffddc9-928xg\" (UID: \"a21aa3c4-4210-4c04-be28-16e0b6ffa6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg"
Oct 30 04:41:46.999437 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.998583    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-maas-api-tls\") pod \"maas-api-e2e-explicit-20a650-666dffddc9-928xg\" (UID: \"a21aa3c4-4210-4c04-be28-16e0b6ffa6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg"
Oct 30 04:41:46.999437 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.998650    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpwvg\" (UniqueName: \"kubernetes.io/projected/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-kube-api-access-cpwvg\") pod \"maas-api-e2e-explicit-20a650-666dffddc9-928xg\" (UID: \"a21aa3c4-4210-4c04-be28-16e0b6ffa6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg"
Oct 30 04:41:46.999897 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:46.999868    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-openshift-service-ca\") pod \"maas-api-e2e-explicit-20a650-666dffddc9-928xg\" (UID: \"a21aa3c4-4210-4c04-be28-16e0b6ffa6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg"
Oct 30 04:41:47.005631 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:47.005600    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-maas-api-tls\") pod \"maas-api-e2e-explicit-20a650-666dffddc9-928xg\" (UID: \"a21aa3c4-4210-4c04-be28-16e0b6ffa6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg"
Oct 30 04:41:47.006371 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:47.006322    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-metrics-certs\") pod \"maas-api-e2e-explicit-20a650-666dffddc9-928xg\" (UID: \"a21aa3c4-4210-4c04-be28-16e0b6ffa6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg"
Oct 30 04:41:47.021714 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:47.021687    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpwvg\" (UniqueName: \"kubernetes.io/projected/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-kube-api-access-cpwvg\") pod \"maas-api-e2e-explicit-20a650-666dffddc9-928xg\" (UID: \"a21aa3c4-4210-4c04-be28-16e0b6ffa6ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg"
Oct 30 04:41:47.079840 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:47.079813    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg"
Oct 30 04:41:47.254771 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:47.254735    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz" event={"ID":"fde6ef20-18b2-4615-952e-8de495f731fe","Type":"ContainerStarted","Data":"bbecf76d40321ecb65e5879d73226a56396305b68f8ac400315abba2fa5b4c30"}
Oct 30 04:41:47.254933 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:47.254802    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz" event={"ID":"fde6ef20-18b2-4615-952e-8de495f731fe","Type":"ContainerStarted","Data":"dc9d3cd0bc646b6afad93648b72085a85c95b5c112f232dc1c344ab272da6d51"}
Oct 30 04:41:47.300932 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:47.300871    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz" podStartSLOduration=2.300853681 podStartE2EDuration="2.300853681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:41:47.297478169 +0000 UTC m=+2626.709873742" watchObservedRunningTime="2026-10-30 04:41:47.300853681 +0000 UTC m=+2626.713249255"
Oct 30 04:41:47.384606 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:47.384582    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg"]
Oct 30 04:41:47.386150 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:41:47.386123    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21aa3c4_4210_4c04_be28_16e0b6ffa6ad.slice/crio-20ffeb6e5ccb95a5df43691a59292fb31c0ca5301017951439a053c4a1bb3904 WatchSource:0}: Error finding container 20ffeb6e5ccb95a5df43691a59292fb31c0ca5301017951439a053c4a1bb3904: Status 404 returned error can't find the container with id 20ffeb6e5ccb95a5df43691a59292fb31c0ca5301017951439a053c4a1bb3904
Oct 30 04:41:48.239817 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:48.239782    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:48.245544 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:48.245523    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:48.262128 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:48.262103    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg" event={"ID":"a21aa3c4-4210-4c04-be28-16e0b6ffa6ad","Type":"ContainerStarted","Data":"dd108b340dd7fac364d63e2c7c1b2b0b6dc57b2678cd35dca97dd7f4a511e90e"}
Oct 30 04:41:48.262261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:48.262142    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg" event={"ID":"a21aa3c4-4210-4c04-be28-16e0b6ffa6ad","Type":"ContainerStarted","Data":"20ffeb6e5ccb95a5df43691a59292fb31c0ca5301017951439a053c4a1bb3904"}
Oct 30 04:41:48.262261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:48.262176    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg"
Oct 30 04:41:48.262437 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:48.262427    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:48.263630 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:48.263614    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:41:48.378061 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:48.377971    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg" podStartSLOduration=1.856566997 podStartE2EDuration="2.377925967s" totalImagesPullingTime="521.35897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:41:48.376456673 +0000 UTC m=+2627.788852245" watchObservedRunningTime="2026-10-30 04:41:48.377925967 +0000 UTC m=+2627.790321541"
Oct 30 04:41:50.142085 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:50.142051    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:41:54.273347 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:41:54.273311    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg"
Oct 30 04:41:55.608923 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:41:55.608883    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:42:12.790354 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:12.790321    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj"]
Oct 30 04:42:12.795855 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:12.795831    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj"
Oct 30 04:42:12.812649 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:12.812628    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-bbce72-serving-cert\""
Oct 30 04:42:12.828576 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:12.828546    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj"]
Oct 30 04:42:12.903563 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:12.903522    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7575m\" (UniqueName: \"kubernetes.io/projected/255ff92e-6bed-464a-a995-04cbc315cf99-kube-api-access-7575m\") pod \"maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj\" (UID: \"255ff92e-6bed-464a-a995-04cbc315cf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj"
Oct 30 04:42:12.903718 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:12.903701    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/255ff92e-6bed-464a-a995-04cbc315cf99-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj\" (UID: \"255ff92e-6bed-464a-a995-04cbc315cf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj"
Oct 30 04:42:12.903783 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:12.903761    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/255ff92e-6bed-464a-a995-04cbc315cf99-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj\" (UID: \"255ff92e-6bed-464a-a995-04cbc315cf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj"
Oct 30 04:42:12.903896 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:12.903878    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/255ff92e-6bed-464a-a995-04cbc315cf99-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj\" (UID: \"255ff92e-6bed-464a-a995-04cbc315cf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj"
Oct 30 04:42:13.029260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:13.012868    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/255ff92e-6bed-464a-a995-04cbc315cf99-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj\" (UID: \"255ff92e-6bed-464a-a995-04cbc315cf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj"
Oct 30 04:42:13.029260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:13.012941    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/255ff92e-6bed-464a-a995-04cbc315cf99-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj\" (UID: \"255ff92e-6bed-464a-a995-04cbc315cf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj"
Oct 30 04:42:13.029260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:13.013014    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/255ff92e-6bed-464a-a995-04cbc315cf99-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj\" (UID: \"255ff92e-6bed-464a-a995-04cbc315cf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj"
Oct 30 04:42:13.029260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:13.013118    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7575m\" (UniqueName: \"kubernetes.io/projected/255ff92e-6bed-464a-a995-04cbc315cf99-kube-api-access-7575m\") pod \"maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj\" (UID: \"255ff92e-6bed-464a-a995-04cbc315cf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj"
Oct 30 04:42:13.036108 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:13.030775    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/255ff92e-6bed-464a-a995-04cbc315cf99-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj\" (UID: \"255ff92e-6bed-464a-a995-04cbc315cf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj"
Oct 30 04:42:13.040028 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:13.040000    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/255ff92e-6bed-464a-a995-04cbc315cf99-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj\" (UID: \"255ff92e-6bed-464a-a995-04cbc315cf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj"
Oct 30 04:42:13.043811 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:13.042000    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/255ff92e-6bed-464a-a995-04cbc315cf99-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj\" (UID: \"255ff92e-6bed-464a-a995-04cbc315cf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj"
Oct 30 04:42:13.056010 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:13.055976    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7575m\" (UniqueName: \"kubernetes.io/projected/255ff92e-6bed-464a-a995-04cbc315cf99-kube-api-access-7575m\") pod \"maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj\" (UID: \"255ff92e-6bed-464a-a995-04cbc315cf99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj"
Oct 30 04:42:13.110221 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:13.110195    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj"
Oct 30 04:42:13.412736 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:13.412710    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj"]
Oct 30 04:42:13.414888 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:42:13.414862    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod255ff92e_6bed_464a_a995_04cbc315cf99.slice/crio-9d0915f82e3b810274ae2f8ba8c975c165e9c2500a18acf4bd48459f63cd6935 WatchSource:0}: Error finding container 9d0915f82e3b810274ae2f8ba8c975c165e9c2500a18acf4bd48459f63cd6935: Status 404 returned error can't find the container with id 9d0915f82e3b810274ae2f8ba8c975c165e9c2500a18acf4bd48459f63cd6935
Oct 30 04:42:14.413004 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:14.412973    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj" event={"ID":"255ff92e-6bed-464a-a995-04cbc315cf99","Type":"ContainerStarted","Data":"c6ff7da61618ce5d351e439a45b8c079cecd1bbd0c7f3e9efece40699810d6f4"}
Oct 30 04:42:14.413412 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:14.413021    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj" event={"ID":"255ff92e-6bed-464a-a995-04cbc315cf99","Type":"ContainerStarted","Data":"9d0915f82e3b810274ae2f8ba8c975c165e9c2500a18acf4bd48459f63cd6935"}
Oct 30 04:42:14.413412 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:14.413051    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj"
Oct 30 04:42:14.461663 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:14.456856    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj" podStartSLOduration=2.15170668 podStartE2EDuration="2.456832086s" totalImagesPullingTime="305.125406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:42:14.453785365 +0000 UTC m=+2653.866180938" watchObservedRunningTime="2026-10-30 04:42:14.456832086 +0000 UTC m=+2653.869227659"
Oct 30 04:42:20.424393 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:20.424364    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj"
Oct 30 04:42:25.609274 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:42:25.609217    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:42:41.117522 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:41.117483    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:42:52.692678 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:52.692645    4874 generic.go:356] "Generic (PLEG): container finished" podID="fde6ef20-18b2-4615-952e-8de495f731fe" containerID="bbecf76d40321ecb65e5879d73226a56396305b68f8ac400315abba2fa5b4c30" exitCode=0
Oct 30 04:42:52.693229 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:52.692699    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz" event={"ID":"fde6ef20-18b2-4615-952e-8de495f731fe","Type":"ContainerDied","Data":"bbecf76d40321ecb65e5879d73226a56396305b68f8ac400315abba2fa5b4c30"}
Oct 30 04:42:52.693229 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:52.693133    4874 scope.go:122] "RemoveContainer" containerID="bbecf76d40321ecb65e5879d73226a56396305b68f8ac400315abba2fa5b4c30"
Oct 30 04:42:52.694628 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:52.694595    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:42:52.694730 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:52.694665    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:42:52.694730 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:52.694697    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:42:53.700406 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:53.700371    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz" event={"ID":"fde6ef20-18b2-4615-952e-8de495f731fe","Type":"ContainerStarted","Data":"f1640ddf5b217ccb2197286722e084ad3da4af63fc0df8487c3af383acf9d929"}
Oct 30 04:42:54.238979 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:54.238945    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:42:54.239189 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:54.239085    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 30 04:42:54.239189 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:54.239135    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz" podUID="fde6ef20-18b2-4615-952e-8de495f731fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 30 04:42:55.239539 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:55.239503    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 30 04:42:55.240035 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:55.239573    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz" podUID="fde6ef20-18b2-4615-952e-8de495f731fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 30 04:42:55.614353 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:42:55.614260    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:42:56.240155 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:56.240120    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 30 04:42:56.240570 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:56.240182    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz" podUID="fde6ef20-18b2-4615-952e-8de495f731fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 30 04:42:57.239669 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:57.239633    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 30 04:42:57.239853 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:57.239693    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz" podUID="fde6ef20-18b2-4615-952e-8de495f731fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 30 04:42:58.239183 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:58.239151    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Oct 30 04:42:58.239586 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:58.239202    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz" podUID="fde6ef20-18b2-4615-952e-8de495f731fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Oct 30 04:42:59.244131 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:59.244102    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:42:59.244548 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:59.244387    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:42:59.245031 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:42:59.245015    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 04:43:00.992425 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:00.992401    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 04:43:01.027299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:01.027277    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 04:43:13.024486 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:13.024453    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4"]
Oct 30 04:43:18.025092 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:18.025032    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4" podUID="b37a947d-42b4-4b02-a8bb-17db1cda4b85" containerName="payload-pre-processing" containerID="cri-o://b0aab1513a0adc2e82a7816c6ba52c7fe5e9d2f67242b46685a7678426c655ad" gracePeriod=25
Oct 30 04:43:18.116990 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:43:18.116958    4874 logging.go:55] [core] [Channel #2083 SubChannel #2084]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.133:50052", ServerName: "10.134.0.133:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.133:50052: connect: connection refused"
Oct 30 04:43:18.292339 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:18.292314    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4"
Oct 30 04:43:18.334690 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:18.334665    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37a947d-42b4-4b02-a8bb-17db1cda4b85-kube-api-access-wzdrf\" (UniqueName: \"kubernetes.io/projected/b37a947d-42b4-4b02-a8bb-17db1cda4b85-kube-api-access-wzdrf\") pod \"b37a947d-42b4-4b02-a8bb-17db1cda4b85\" (UID: \"b37a947d-42b4-4b02-a8bb-17db1cda4b85\") "
Oct 30 04:43:18.334835 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:18.334811    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b37a947d-42b4-4b02-a8bb-17db1cda4b85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37a947d-42b4-4b02-a8bb-17db1cda4b85-plugins-config-volume\") pod \"b37a947d-42b4-4b02-a8bb-17db1cda4b85\" (UID: \"b37a947d-42b4-4b02-a8bb-17db1cda4b85\") "
Oct 30 04:43:18.335175 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:18.335149    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b37a947d-42b4-4b02-a8bb-17db1cda4b85-plugins-config-volume" pod "b37a947d-42b4-4b02-a8bb-17db1cda4b85" (UID: "b37a947d-42b4-4b02-a8bb-17db1cda4b85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:43:18.337206 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:18.337180    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37a947d-42b4-4b02-a8bb-17db1cda4b85-kube-api-access-wzdrf" pod "b37a947d-42b4-4b02-a8bb-17db1cda4b85" (UID: "b37a947d-42b4-4b02-a8bb-17db1cda4b85"). InnerVolumeSpecName "kube-api-access-wzdrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:43:18.435888 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:18.435857    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37a947d-42b4-4b02-a8bb-17db1cda4b85-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:43:18.435888 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:18.435888    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzdrf\" (UniqueName: \"kubernetes.io/projected/b37a947d-42b4-4b02-a8bb-17db1cda4b85-kube-api-access-wzdrf\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:43:18.884675 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:18.884643    4874 generic.go:356] "Generic (PLEG): container finished" podID="b37a947d-42b4-4b02-a8bb-17db1cda4b85" containerID="b0aab1513a0adc2e82a7816c6ba52c7fe5e9d2f67242b46685a7678426c655ad" exitCode=0
Oct 30 04:43:18.884881 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:18.884690    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4"
Oct 30 04:43:18.884881 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:18.884713    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4" event={"ID":"b37a947d-42b4-4b02-a8bb-17db1cda4b85","Type":"ContainerDied","Data":"b0aab1513a0adc2e82a7816c6ba52c7fe5e9d2f67242b46685a7678426c655ad"}
Oct 30 04:43:18.884881 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:18.884754    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4" event={"ID":"b37a947d-42b4-4b02-a8bb-17db1cda4b85","Type":"ContainerDied","Data":"99a5ecdedc03822c06b9b5b6237dc55a28e4ec06c88ad09fba83e94163149fd9"}
Oct 30 04:43:18.884881 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:18.884775    4874 scope.go:122] "RemoveContainer" containerID="b0aab1513a0adc2e82a7816c6ba52c7fe5e9d2f67242b46685a7678426c655ad"
Oct 30 04:43:18.893970 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:18.893951    4874 scope.go:122] "RemoveContainer" containerID="b0aab1513a0adc2e82a7816c6ba52c7fe5e9d2f67242b46685a7678426c655ad"
Oct 30 04:43:18.894230 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:43:18.894211    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0aab1513a0adc2e82a7816c6ba52c7fe5e9d2f67242b46685a7678426c655ad\": container with ID starting with b0aab1513a0adc2e82a7816c6ba52c7fe5e9d2f67242b46685a7678426c655ad not found: ID does not exist" containerID="b0aab1513a0adc2e82a7816c6ba52c7fe5e9d2f67242b46685a7678426c655ad"
Oct 30 04:43:18.894307 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:18.894249    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0aab1513a0adc2e82a7816c6ba52c7fe5e9d2f67242b46685a7678426c655ad"} err="failed to get container status \"b0aab1513a0adc2e82a7816c6ba52c7fe5e9d2f67242b46685a7678426c655ad\": rpc error: code = NotFound desc = could not find container \"b0aab1513a0adc2e82a7816c6ba52c7fe5e9d2f67242b46685a7678426c655ad\": container with ID starting with b0aab1513a0adc2e82a7816c6ba52c7fe5e9d2f67242b46685a7678426c655ad not found: ID does not exist"
Oct 30 04:43:18.934316 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:18.934288    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4"]
Oct 30 04:43:18.945388 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:18.945363    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4"]
Oct 30 04:43:19.117771 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:19.117682    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-explicit-20a650-775687f6b7-r68v4" podUID="b37a947d-42b4-4b02-a8bb-17db1cda4b85" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.133:50052\" within 1s: context deadline exceeded"
Oct 30 04:43:19.117771 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:43:19.117714    4874 logging.go:55] [core] [Channel #2083 SubChannel #2084]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.133:50052", ServerName: "10.134.0.133:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.133:50052: operation was canceled"
Oct 30 04:43:20.729380 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:20.729345    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b37a947d-42b4-4b02-a8bb-17db1cda4b85" path="/var/lib/kubelet/pods/b37a947d-42b4-4b02-a8bb-17db1cda4b85/volumes"
Oct 30 04:43:25.610267 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:43:25.610214    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:43:27.068488 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:27.068455    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:43:27.847491 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:27.847452    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg"]
Oct 30 04:43:27.848052 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:27.847999    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg" podUID="a21aa3c4-4210-4c04-be28-16e0b6ffa6ad" containerName="maas-api" containerID="cri-o://dd108b340dd7fac364d63e2c7c1b2b0b6dc57b2678cd35dca97dd7f4a511e90e" gracePeriod=30
Oct 30 04:43:28.329799 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:28.329499    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg"
Oct 30 04:43:28.491170 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:28.491135    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-openshift-service-ca\") pod \"a21aa3c4-4210-4c04-be28-16e0b6ffa6ad\" (UID: \"a21aa3c4-4210-4c04-be28-16e0b6ffa6ad\") "
Oct 30 04:43:28.491508 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:28.491481    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-maas-api-tls\") pod \"a21aa3c4-4210-4c04-be28-16e0b6ffa6ad\" (UID: \"a21aa3c4-4210-4c04-be28-16e0b6ffa6ad\") "
Oct 30 04:43:28.491716 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:28.491695    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-metrics-certs\") pod \"a21aa3c4-4210-4c04-be28-16e0b6ffa6ad\" (UID: \"a21aa3c4-4210-4c04-be28-16e0b6ffa6ad\") "
Oct 30 04:43:28.491862 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:28.491845    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-kube-api-access-cpwvg\" (UniqueName: \"kubernetes.io/projected/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-kube-api-access-cpwvg\") pod \"a21aa3c4-4210-4c04-be28-16e0b6ffa6ad\" (UID: \"a21aa3c4-4210-4c04-be28-16e0b6ffa6ad\") "
Oct 30 04:43:28.494740 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:28.494713    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-openshift-service-ca" pod "a21aa3c4-4210-4c04-be28-16e0b6ffa6ad" (UID: "a21aa3c4-4210-4c04-be28-16e0b6ffa6ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:43:28.501701 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:28.501653    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-metrics-certs" pod "a21aa3c4-4210-4c04-be28-16e0b6ffa6ad" (UID: "a21aa3c4-4210-4c04-be28-16e0b6ffa6ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:43:28.501902 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:28.501788    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-maas-api-tls" pod "a21aa3c4-4210-4c04-be28-16e0b6ffa6ad" (UID: "a21aa3c4-4210-4c04-be28-16e0b6ffa6ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:43:28.502439 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:28.502411    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-kube-api-access-cpwvg" pod "a21aa3c4-4210-4c04-be28-16e0b6ffa6ad" (UID: "a21aa3c4-4210-4c04-be28-16e0b6ffa6ad"). InnerVolumeSpecName "kube-api-access-cpwvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:43:28.599534 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:28.599440    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:43:28.599769 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:28.599748    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:43:28.599902 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:28.599889    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:43:28.600020 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:28.600005    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpwvg\" (UniqueName: \"kubernetes.io/projected/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad-kube-api-access-cpwvg\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:43:28.944479 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:28.944397    4874 generic.go:356] "Generic (PLEG): container finished" podID="a21aa3c4-4210-4c04-be28-16e0b6ffa6ad" containerID="dd108b340dd7fac364d63e2c7c1b2b0b6dc57b2678cd35dca97dd7f4a511e90e" exitCode=0
Oct 30 04:43:28.944479 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:28.944446    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg"
Oct 30 04:43:28.944656 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:28.944491    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg" event={"ID":"a21aa3c4-4210-4c04-be28-16e0b6ffa6ad","Type":"ContainerDied","Data":"dd108b340dd7fac364d63e2c7c1b2b0b6dc57b2678cd35dca97dd7f4a511e90e"}
Oct 30 04:43:28.944656 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:28.944566    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg" event={"ID":"a21aa3c4-4210-4c04-be28-16e0b6ffa6ad","Type":"ContainerDied","Data":"20ffeb6e5ccb95a5df43691a59292fb31c0ca5301017951439a053c4a1bb3904"}
Oct 30 04:43:28.944656 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:28.944598    4874 scope.go:122] "RemoveContainer" containerID="dd108b340dd7fac364d63e2c7c1b2b0b6dc57b2678cd35dca97dd7f4a511e90e"
Oct 30 04:43:28.954424 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:28.954319    4874 scope.go:122] "RemoveContainer" containerID="dd108b340dd7fac364d63e2c7c1b2b0b6dc57b2678cd35dca97dd7f4a511e90e"
Oct 30 04:43:28.954637 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:43:28.954609    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd108b340dd7fac364d63e2c7c1b2b0b6dc57b2678cd35dca97dd7f4a511e90e\": container with ID starting with dd108b340dd7fac364d63e2c7c1b2b0b6dc57b2678cd35dca97dd7f4a511e90e not found: ID does not exist" containerID="dd108b340dd7fac364d63e2c7c1b2b0b6dc57b2678cd35dca97dd7f4a511e90e"
Oct 30 04:43:28.954690 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:28.954644    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd108b340dd7fac364d63e2c7c1b2b0b6dc57b2678cd35dca97dd7f4a511e90e"} err="failed to get container status \"dd108b340dd7fac364d63e2c7c1b2b0b6dc57b2678cd35dca97dd7f4a511e90e\": rpc error: code = NotFound desc = could not find container \"dd108b340dd7fac364d63e2c7c1b2b0b6dc57b2678cd35dca97dd7f4a511e90e\": container with ID starting with dd108b340dd7fac364d63e2c7c1b2b0b6dc57b2678cd35dca97dd7f4a511e90e not found: ID does not exist"
Oct 30 04:43:28.997498 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:28.997469    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg"]
Oct 30 04:43:29.007634 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:29.007601    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-20a650-666dffddc9-928xg"]
Oct 30 04:43:30.729656 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:30.729624    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a21aa3c4-4210-4c04-be28-16e0b6ffa6ad" path="/var/lib/kubelet/pods/a21aa3c4-4210-4c04-be28-16e0b6ffa6ad/volumes"
Oct 30 04:43:37.378439 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.378408    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"]
Oct 30 04:43:37.378971 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.378951    4874 cpu_manager.go:414] "containerMap: removing container" podUID="b37a947d-42b4-4b02-a8bb-17db1cda4b85" containerName="payload-pre-processing"
Oct 30 04:43:37.379015 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.378972    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b37a947d-42b4-4b02-a8bb-17db1cda4b85" containerName="payload-pre-processing"
Oct 30 04:43:37.379015 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.378996    4874 cpu_manager.go:414] "containerMap: removing container" podUID="a21aa3c4-4210-4c04-be28-16e0b6ffa6ad" containerName="maas-api"
Oct 30 04:43:37.379015 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.379001    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a21aa3c4-4210-4c04-be28-16e0b6ffa6ad" containerName="maas-api"
Oct 30 04:43:37.379118 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.379091    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="b37a947d-42b4-4b02-a8bb-17db1cda4b85" containerName="payload-pre-processing"
Oct 30 04:43:37.379118 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.379103    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="a21aa3c4-4210-4c04-be28-16e0b6ffa6ad" containerName="maas-api"
Oct 30 04:43:37.382583 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.382558    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.397215 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.397193    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-ea1940-openshift-default-dockercfg-wg6pj\""
Oct 30 04:43:37.411506 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.411483    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"]
Oct 30 04:43:37.496728 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.496688    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afcb33d7-d878-4994-abda-5e01c0079c2b-istiod-ca-cert\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.497288 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.497233    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-data\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.497844 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.497818    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-envoy\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.498166 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.498147    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-workload-certs\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.498479 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.498388    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-workload-socket\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.499319 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.498864    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh8bp\" (UniqueName: \"kubernetes.io/projected/afcb33d7-d878-4994-abda-5e01c0079c2b-kube-api-access-fh8bp\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.499547 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.499527    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-credential-socket\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.499783 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.499766    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-token\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.500024 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.500007    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-podinfo\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.601732 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.601689    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afcb33d7-d878-4994-abda-5e01c0079c2b-istiod-ca-cert\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.601951 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.601784    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-data\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.601951 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.601826    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-envoy\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.601951 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.601872    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-workload-certs\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.601951 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.601899    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-workload-socket\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.601951 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.601930    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh8bp\" (UniqueName: \"kubernetes.io/projected/afcb33d7-d878-4994-abda-5e01c0079c2b-kube-api-access-fh8bp\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.602345 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.601966    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-credential-socket\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.602345 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.602014    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-token\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.602345 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.602072    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-podinfo\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.602637 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.602615    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afcb33d7-d878-4994-abda-5e01c0079c2b-istiod-ca-cert\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.602861 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.602840    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-workload-socket\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.603027 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.603012    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-data\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.603454 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.603428    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-credential-socket\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.603975 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.603954    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-workload-certs\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.605530 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.605506    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-podinfo\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.608713 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.608691    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-envoy\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.637092 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.637027    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-token\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.637292 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.637273    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh8bp\" (UniqueName: \"kubernetes.io/projected/afcb33d7-d878-4994-abda-5e01c0079c2b-kube-api-access-fh8bp\") pod \"e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") " pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:37.696629 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:37.696598    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:38.026716 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:38.026690    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"]
Oct 30 04:43:38.029586 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:43:38.029559    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafcb33d7_d878_4994_abda_5e01c0079c2b.slice/crio-71b56190e081ac980e81172535edd8e9b9f571fdea5bdc756fc4fb1c55cdda42 WatchSource:0}: Error finding container 71b56190e081ac980e81172535edd8e9b9f571fdea5bdc756fc4fb1c55cdda42: Status 404 returned error can't find the container with id 71b56190e081ac980e81172535edd8e9b9f571fdea5bdc756fc4fb1c55cdda42
Oct 30 04:43:38.031842 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:38.031807    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:43:38.031954 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:38.031893    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:43:38.031954 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:38.031936    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:43:39.009400 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:39.009362    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t" event={"ID":"afcb33d7-d878-4994-abda-5e01c0079c2b","Type":"ContainerStarted","Data":"afad65b89c4c5782559d0ae25d21ce321a12a79b78332724ef93976500ac75f6"}
Oct 30 04:43:39.009793 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:39.009413    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t" event={"ID":"afcb33d7-d878-4994-abda-5e01c0079c2b","Type":"ContainerStarted","Data":"71b56190e081ac980e81172535edd8e9b9f571fdea5bdc756fc4fb1c55cdda42"}
Oct 30 04:43:39.067126 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:39.067065    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t" podStartSLOduration=2.067049389 podStartE2EDuration="2.067049389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:39.065378964 +0000 UTC m=+2738.477774563" watchObservedRunningTime="2026-10-30 04:43:39.067049389 +0000 UTC m=+2738.479444969"
Oct 30 04:43:39.697460 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:39.697427    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:39.703485 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:39.703458    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:40.014646 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:40.014621    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:40.015556 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:40.015533    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:43:42.061569 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:42.061534    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"]
Oct 30 04:43:42.062064 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:42.061772    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz" podUID="ee079d02-8eaf-4b69-9f04-ffdb2f5073a2" containerName="istio-proxy" containerID="cri-o://c8f8d9f2a6b1b86b0beb50dbf66d75de521ece8dfeba5dbda7f6a1e8cde905aa" gracePeriod=30
Oct 30 04:43:47.311204 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.311180    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:43:47.450825 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.450754    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-token\" (UniqueName: \"kubernetes.io/projected/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-token\") pod \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") "
Oct 30 04:43:47.450825 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.450798    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-workload-socket\") pod \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") "
Oct 30 04:43:47.451064 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.450828    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-credential-socket\") pod \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") "
Oct 30 04:43:47.451064 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.450952    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-data\") pod \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") "
Oct 30 04:43:47.451064 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.451028    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-workload-socket" pod "ee079d02-8eaf-4b69-9f04-ffdb2f5073a2" (UID: "ee079d02-8eaf-4b69-9f04-ffdb2f5073a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:43:47.451064 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.451050    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-kube-api-access-lmkcq\" (UniqueName: \"kubernetes.io/projected/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-kube-api-access-lmkcq\") pod \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") "
Oct 30 04:43:47.451313 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.451119    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-workload-certs\") pod \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") "
Oct 30 04:43:47.451313 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.451134    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-credential-socket" pod "ee079d02-8eaf-4b69-9f04-ffdb2f5073a2" (UID: "ee079d02-8eaf-4b69-9f04-ffdb2f5073a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:43:47.451313 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.451165    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istiod-ca-cert\") pod \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") "
Oct 30 04:43:47.451313 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.451203    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-data" pod "ee079d02-8eaf-4b69-9f04-ffdb2f5073a2" (UID: "ee079d02-8eaf-4b69-9f04-ffdb2f5073a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:43:47.451313 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.451207    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-envoy\") pod \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") "
Oct 30 04:43:47.451313 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.451298    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-podinfo\") pod \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\" (UID: \"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2\") "
Oct 30 04:43:47.451653 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.451397    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-workload-certs" pod "ee079d02-8eaf-4b69-9f04-ffdb2f5073a2" (UID: "ee079d02-8eaf-4b69-9f04-ffdb2f5073a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:43:47.451653 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.451538    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istiod-ca-cert" pod "ee079d02-8eaf-4b69-9f04-ffdb2f5073a2" (UID: "ee079d02-8eaf-4b69-9f04-ffdb2f5073a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:43:47.451989 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.451965    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:43:47.452042 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.451997    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:43:47.452042 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.452013    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:43:47.452042 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.452028    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:43:47.452150 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.452041    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:43:47.453285 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.453250    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-kube-api-access-lmkcq" pod "ee079d02-8eaf-4b69-9f04-ffdb2f5073a2" (UID: "ee079d02-8eaf-4b69-9f04-ffdb2f5073a2"). InnerVolumeSpecName "kube-api-access-lmkcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:43:47.453285 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.453275    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-envoy" pod "ee079d02-8eaf-4b69-9f04-ffdb2f5073a2" (UID: "ee079d02-8eaf-4b69-9f04-ffdb2f5073a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:43:47.453478 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.453458    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-token" pod "ee079d02-8eaf-4b69-9f04-ffdb2f5073a2" (UID: "ee079d02-8eaf-4b69-9f04-ffdb2f5073a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:43:47.453580 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.453563    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-podinfo" pod "ee079d02-8eaf-4b69-9f04-ffdb2f5073a2" (UID: "ee079d02-8eaf-4b69-9f04-ffdb2f5073a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:43:47.553434 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.553399    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:43:47.553434 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.553427    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmkcq\" (UniqueName: \"kubernetes.io/projected/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-kube-api-access-lmkcq\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:43:47.553434 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.553440    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:43:47.553663 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:47.553452    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:43:48.059675 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:48.059638    4874 generic.go:356] "Generic (PLEG): container finished" podID="ee079d02-8eaf-4b69-9f04-ffdb2f5073a2" containerID="c8f8d9f2a6b1b86b0beb50dbf66d75de521ece8dfeba5dbda7f6a1e8cde905aa" exitCode=0
Oct 30 04:43:48.059851 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:48.059691    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"
Oct 30 04:43:48.059851 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:48.059724    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz" event={"ID":"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2","Type":"ContainerDied","Data":"c8f8d9f2a6b1b86b0beb50dbf66d75de521ece8dfeba5dbda7f6a1e8cde905aa"}
Oct 30 04:43:48.059851 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:48.059787    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz" event={"ID":"ee079d02-8eaf-4b69-9f04-ffdb2f5073a2","Type":"ContainerDied","Data":"d40f22c63fc4020c549d877025ae26c34a3937c2fa41d4be1ce7d7d8bd0e1dd7"}
Oct 30 04:43:48.059851 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:48.059814    4874 scope.go:122] "RemoveContainer" containerID="c8f8d9f2a6b1b86b0beb50dbf66d75de521ece8dfeba5dbda7f6a1e8cde905aa"
Oct 30 04:43:48.069594 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:48.069576    4874 scope.go:122] "RemoveContainer" containerID="c8f8d9f2a6b1b86b0beb50dbf66d75de521ece8dfeba5dbda7f6a1e8cde905aa"
Oct 30 04:43:48.069839 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:43:48.069822    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f8d9f2a6b1b86b0beb50dbf66d75de521ece8dfeba5dbda7f6a1e8cde905aa\": container with ID starting with c8f8d9f2a6b1b86b0beb50dbf66d75de521ece8dfeba5dbda7f6a1e8cde905aa not found: ID does not exist" containerID="c8f8d9f2a6b1b86b0beb50dbf66d75de521ece8dfeba5dbda7f6a1e8cde905aa"
Oct 30 04:43:48.069896 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:48.069845    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f8d9f2a6b1b86b0beb50dbf66d75de521ece8dfeba5dbda7f6a1e8cde905aa"} err="failed to get container status \"c8f8d9f2a6b1b86b0beb50dbf66d75de521ece8dfeba5dbda7f6a1e8cde905aa\": rpc error: code = NotFound desc = could not find container \"c8f8d9f2a6b1b86b0beb50dbf66d75de521ece8dfeba5dbda7f6a1e8cde905aa\": container with ID starting with c8f8d9f2a6b1b86b0beb50dbf66d75de521ece8dfeba5dbda7f6a1e8cde905aa not found: ID does not exist"
Oct 30 04:43:48.109874 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:48.109849    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"]
Oct 30 04:43:48.122279 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:48.122232    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-wsnwz"]
Oct 30 04:43:48.739083 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:48.739049    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee079d02-8eaf-4b69-9f04-ffdb2f5073a2" path="/var/lib/kubelet/pods/ee079d02-8eaf-4b69-9f04-ffdb2f5073a2/volumes"
Oct 30 04:43:55.615543 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:43:55.615507    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:43:56.694156 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.694122    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"]
Oct 30 04:43:56.694949 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.694927    4874 cpu_manager.go:414] "containerMap: removing container" podUID="ee079d02-8eaf-4b69-9f04-ffdb2f5073a2" containerName="istio-proxy"
Oct 30 04:43:56.695000 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.694953    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee079d02-8eaf-4b69-9f04-ffdb2f5073a2" containerName="istio-proxy"
Oct 30 04:43:56.695158 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.695145    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee079d02-8eaf-4b69-9f04-ffdb2f5073a2" containerName="istio-proxy"
Oct 30 04:43:56.701075 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.701052    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.719823 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.719786    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"]
Oct 30 04:43:56.727470 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.727442    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-ed8266-openshift-default-dockercfg-cmnbn\""
Oct 30 04:43:56.753700 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.753666    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06eecd27-740b-48d7-903c-182db0dae7ad-istio-token\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.753866 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.753772    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-istio-data\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.753866 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.753828    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-istio-envoy\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.753866 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.753858    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-workload-socket\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.754043 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.753975    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-credential-socket\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.754043 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.754037    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06eecd27-740b-48d7-903c-182db0dae7ad-istiod-ca-cert\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.754182 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.754154    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbkhl\" (UniqueName: \"kubernetes.io/projected/06eecd27-740b-48d7-903c-182db0dae7ad-kube-api-access-mbkhl\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.754339 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.754231    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06eecd27-740b-48d7-903c-182db0dae7ad-istio-podinfo\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.754339 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.754298    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-workload-certs\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.868282 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.865227    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbkhl\" (UniqueName: \"kubernetes.io/projected/06eecd27-740b-48d7-903c-182db0dae7ad-kube-api-access-mbkhl\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.868282 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.865343    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06eecd27-740b-48d7-903c-182db0dae7ad-istio-podinfo\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.868282 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.865386    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-workload-certs\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.868282 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.865466    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06eecd27-740b-48d7-903c-182db0dae7ad-istio-token\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.868282 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.865521    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-istio-data\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.868282 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.865598    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-istio-envoy\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.868282 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.865637    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-workload-socket\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.868282 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.865676    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-credential-socket\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.868282 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.865901    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06eecd27-740b-48d7-903c-182db0dae7ad-istiod-ca-cert\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.868282 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.866846    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06eecd27-740b-48d7-903c-182db0dae7ad-istiod-ca-cert\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.868975 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.868525    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-istio-data\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.868975 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.868779    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-workload-certs\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.869354 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.869331    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-workload-socket\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.871614 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.870635    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-credential-socket\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.874616 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.873889    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06eecd27-740b-48d7-903c-182db0dae7ad-istio-podinfo\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.875315 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.875294    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-istio-envoy\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.893138 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.893110    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbkhl\" (UniqueName: \"kubernetes.io/projected/06eecd27-740b-48d7-903c-182db0dae7ad-kube-api-access-mbkhl\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:56.896794 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:56.896768    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06eecd27-740b-48d7-903c-182db0dae7ad-istio-token\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:57.016553 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:57.016519    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:57.339396 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:57.339369    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"]
Oct 30 04:43:57.341227 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:43:57.341198    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06eecd27_740b_48d7_903c_182db0dae7ad.slice/crio-62f3db08928d003e7170f0c3d45e05706e4f4cf2e3437768c1f4238e356245ed WatchSource:0}: Error finding container 62f3db08928d003e7170f0c3d45e05706e4f4cf2e3437768c1f4238e356245ed: Status 404 returned error can't find the container with id 62f3db08928d003e7170f0c3d45e05706e4f4cf2e3437768c1f4238e356245ed
Oct 30 04:43:57.343745 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:57.343709    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:43:57.343837 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:57.343774    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:43:57.343837 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:57.343803    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:43:58.136651 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:58.136615    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb" event={"ID":"06eecd27-740b-48d7-903c-182db0dae7ad","Type":"ContainerStarted","Data":"8163ae0c23b5fbf2851e182dbbad87efd76e629599cfc1623582f8c765610220"}
Oct 30 04:43:58.137066 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:58.136672    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb" event={"ID":"06eecd27-740b-48d7-903c-182db0dae7ad","Type":"ContainerStarted","Data":"62f3db08928d003e7170f0c3d45e05706e4f4cf2e3437768c1f4238e356245ed"}
Oct 30 04:43:58.192168 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:58.192120    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb" podStartSLOduration=2.192103148 podStartE2EDuration="2.192103148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:43:58.191341953 +0000 UTC m=+2757.603737525" watchObservedRunningTime="2026-10-30 04:43:58.192103148 +0000 UTC m=+2757.604498723"
Oct 30 04:43:59.016969 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:59.016936    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:59.022498 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:59.022476    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:59.143787 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:59.143753    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:43:59.144756 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:43:59.144737    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:44:05.123531 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:05.123495    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"]
Oct 30 04:44:05.124034 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:05.123801    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t" podUID="afcb33d7-d878-4994-abda-5e01c0079c2b" containerName="istio-proxy" containerID="cri-o://afad65b89c4c5782559d0ae25d21ce321a12a79b78332724ef93976500ac75f6" gracePeriod=30
Oct 30 04:44:10.017307 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.017270    4874 patch_prober.go:29] interesting pod/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 04:44:10.017826 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.017334    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t" podUID="afcb33d7-d878-4994-abda-5e01c0079c2b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:44:10.205498 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.205466    4874 generic.go:356] "Generic (PLEG): container finished" podID="afcb33d7-d878-4994-abda-5e01c0079c2b" containerID="afad65b89c4c5782559d0ae25d21ce321a12a79b78332724ef93976500ac75f6" exitCode=0
Oct 30 04:44:10.205625 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.205534    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t" event={"ID":"afcb33d7-d878-4994-abda-5e01c0079c2b","Type":"ContainerDied","Data":"afad65b89c4c5782559d0ae25d21ce321a12a79b78332724ef93976500ac75f6"}
Oct 30 04:44:10.787748 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.787725    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:44:10.937728 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.937645    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-workload-certs\") pod \"afcb33d7-d878-4994-abda-5e01c0079c2b\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") "
Oct 30 04:44:10.937728 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.937724    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-envoy\") pod \"afcb33d7-d878-4994-abda-5e01c0079c2b\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") "
Oct 30 04:44:10.937968 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.937756    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afcb33d7-d878-4994-abda-5e01c0079c2b-kube-api-access-fh8bp\" (UniqueName: \"kubernetes.io/projected/afcb33d7-d878-4994-abda-5e01c0079c2b-kube-api-access-fh8bp\") pod \"afcb33d7-d878-4994-abda-5e01c0079c2b\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") "
Oct 30 04:44:10.937968 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.937775    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-token\" (UniqueName: \"kubernetes.io/projected/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-token\") pod \"afcb33d7-d878-4994-abda-5e01c0079c2b\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") "
Oct 30 04:44:10.937968 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.937796    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-podinfo\") pod \"afcb33d7-d878-4994-abda-5e01c0079c2b\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") "
Oct 30 04:44:10.937968 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.937946    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-data\") pod \"afcb33d7-d878-4994-abda-5e01c0079c2b\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") "
Oct 30 04:44:10.938193 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.937988    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-workload-certs" pod "afcb33d7-d878-4994-abda-5e01c0079c2b" (UID: "afcb33d7-d878-4994-abda-5e01c0079c2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:44:10.938193 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.938001    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-workload-socket\") pod \"afcb33d7-d878-4994-abda-5e01c0079c2b\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") "
Oct 30 04:44:10.938193 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.938029    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-credential-socket\") pod \"afcb33d7-d878-4994-abda-5e01c0079c2b\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") "
Oct 30 04:44:10.938193 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.938065    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afcb33d7-d878-4994-abda-5e01c0079c2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afcb33d7-d878-4994-abda-5e01c0079c2b-istiod-ca-cert\") pod \"afcb33d7-d878-4994-abda-5e01c0079c2b\" (UID: \"afcb33d7-d878-4994-abda-5e01c0079c2b\") "
Oct 30 04:44:10.938428 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.938228    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-data" pod "afcb33d7-d878-4994-abda-5e01c0079c2b" (UID: "afcb33d7-d878-4994-abda-5e01c0079c2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:44:10.938428 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.938348    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-workload-socket" pod "afcb33d7-d878-4994-abda-5e01c0079c2b" (UID: "afcb33d7-d878-4994-abda-5e01c0079c2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:44:10.938428 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.938409    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-credential-socket" pod "afcb33d7-d878-4994-abda-5e01c0079c2b" (UID: "afcb33d7-d878-4994-abda-5e01c0079c2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:44:10.938671 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.938649    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afcb33d7-d878-4994-abda-5e01c0079c2b-istiod-ca-cert" pod "afcb33d7-d878-4994-abda-5e01c0079c2b" (UID: "afcb33d7-d878-4994-abda-5e01c0079c2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:44:10.938735 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.938701    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:44:10.938735 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.938724    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:44:10.938817 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.938737    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:44:10.938817 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.938753    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:44:10.940168 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.940139    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-podinfo" pod "afcb33d7-d878-4994-abda-5e01c0079c2b" (UID: "afcb33d7-d878-4994-abda-5e01c0079c2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:44:10.940168 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.940148    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afcb33d7-d878-4994-abda-5e01c0079c2b-kube-api-access-fh8bp" pod "afcb33d7-d878-4994-abda-5e01c0079c2b" (UID: "afcb33d7-d878-4994-abda-5e01c0079c2b"). InnerVolumeSpecName "kube-api-access-fh8bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:44:10.940449 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.940429    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-envoy" pod "afcb33d7-d878-4994-abda-5e01c0079c2b" (UID: "afcb33d7-d878-4994-abda-5e01c0079c2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:44:10.940513 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:10.940434    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-token" pod "afcb33d7-d878-4994-abda-5e01c0079c2b" (UID: "afcb33d7-d878-4994-abda-5e01c0079c2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:44:11.039440 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:11.039412    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:44:11.039440 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:11.039436    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh8bp\" (UniqueName: \"kubernetes.io/projected/afcb33d7-d878-4994-abda-5e01c0079c2b-kube-api-access-fh8bp\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:44:11.039811 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:11.039450    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:44:11.039811 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:11.039461    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afcb33d7-d878-4994-abda-5e01c0079c2b-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:44:11.039811 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:11.039470    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afcb33d7-d878-4994-abda-5e01c0079c2b-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:44:11.212933 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:11.212860    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"
Oct 30 04:44:11.212933 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:11.212867    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t" event={"ID":"afcb33d7-d878-4994-abda-5e01c0079c2b","Type":"ContainerDied","Data":"71b56190e081ac980e81172535edd8e9b9f571fdea5bdc756fc4fb1c55cdda42"}
Oct 30 04:44:11.212933 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:11.212928    4874 scope.go:122] "RemoveContainer" containerID="afad65b89c4c5782559d0ae25d21ce321a12a79b78332724ef93976500ac75f6"
Oct 30 04:44:11.262475 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:11.262448    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"]
Oct 30 04:44:11.273874 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:11.273850    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-ea1940-openshift-default-698c8f66c7-qjs9t"]
Oct 30 04:44:12.729316 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:12.729285    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afcb33d7-d878-4994-abda-5e01c0079c2b" path="/var/lib/kubelet/pods/afcb33d7-d878-4994-abda-5e01c0079c2b/volumes"
Oct 30 04:44:17.385474 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:17.385439    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:44:23.683186 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:23.683156    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm"]
Oct 30 04:44:23.683738 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:23.683722    4874 cpu_manager.go:414] "containerMap: removing container" podUID="afcb33d7-d878-4994-abda-5e01c0079c2b" containerName="istio-proxy"
Oct 30 04:44:23.683784 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:23.683739    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afcb33d7-d878-4994-abda-5e01c0079c2b" containerName="istio-proxy"
Oct 30 04:44:23.683848 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:23.683837    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="afcb33d7-d878-4994-abda-5e01c0079c2b" containerName="istio-proxy"
Oct 30 04:44:23.688311 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:23.688290    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm"
Oct 30 04:44:23.696209 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:23.696183    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bea58a5-bdac-4545-9aee-66961d2bd86b-metrics-certs\") pod \"maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm\" (UID: \"2bea58a5-bdac-4545-9aee-66961d2bd86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm"
Oct 30 04:44:23.696361 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:23.696253    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bea58a5-bdac-4545-9aee-66961d2bd86b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm\" (UID: \"2bea58a5-bdac-4545-9aee-66961d2bd86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm"
Oct 30 04:44:23.696361 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:23.696325    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bea58a5-bdac-4545-9aee-66961d2bd86b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm\" (UID: \"2bea58a5-bdac-4545-9aee-66961d2bd86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm"
Oct 30 04:44:23.696486 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:23.696370    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr48c\" (UniqueName: \"kubernetes.io/projected/2bea58a5-bdac-4545-9aee-66961d2bd86b-kube-api-access-gr48c\") pod \"maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm\" (UID: \"2bea58a5-bdac-4545-9aee-66961d2bd86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm"
Oct 30 04:44:23.702278 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:23.702248    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-ed8266-serving-cert\""
Oct 30 04:44:23.717965 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:23.717913    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm"]
Oct 30 04:44:23.797284 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:23.797222    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bea58a5-bdac-4545-9aee-66961d2bd86b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm\" (UID: \"2bea58a5-bdac-4545-9aee-66961d2bd86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm"
Oct 30 04:44:23.797476 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:23.797291    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bea58a5-bdac-4545-9aee-66961d2bd86b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm\" (UID: \"2bea58a5-bdac-4545-9aee-66961d2bd86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm"
Oct 30 04:44:23.797476 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:23.797318    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr48c\" (UniqueName: \"kubernetes.io/projected/2bea58a5-bdac-4545-9aee-66961d2bd86b-kube-api-access-gr48c\") pod \"maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm\" (UID: \"2bea58a5-bdac-4545-9aee-66961d2bd86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm"
Oct 30 04:44:23.797476 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:23.797423    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bea58a5-bdac-4545-9aee-66961d2bd86b-metrics-certs\") pod \"maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm\" (UID: \"2bea58a5-bdac-4545-9aee-66961d2bd86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm"
Oct 30 04:44:23.797476 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:44:23.797435    4874 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-serving-cert: secret "maas-api-e2e-extmod-a-ed8266-serving-cert" not found
Oct 30 04:44:23.797723 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:44:23.797511    4874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bea58a5-bdac-4545-9aee-66961d2bd86b-maas-api-tls podName:2bea58a5-bdac-4545-9aee-66961d2bd86b nodeName:}" failed. No retries permitted until 2026-10-30 04:44:24.297495707 +0000 UTC m=+2783.709891273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2bea58a5-bdac-4545-9aee-66961d2bd86b-maas-api-tls") pod "maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm" (UID: "2bea58a5-bdac-4545-9aee-66961d2bd86b") : secret "maas-api-e2e-extmod-a-ed8266-serving-cert" not found
Oct 30 04:44:23.797970 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:23.797947    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bea58a5-bdac-4545-9aee-66961d2bd86b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm\" (UID: \"2bea58a5-bdac-4545-9aee-66961d2bd86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm"
Oct 30 04:44:23.799791 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:23.799770    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bea58a5-bdac-4545-9aee-66961d2bd86b-metrics-certs\") pod \"maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm\" (UID: \"2bea58a5-bdac-4545-9aee-66961d2bd86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm"
Oct 30 04:44:23.819869 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:23.819842    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr48c\" (UniqueName: \"kubernetes.io/projected/2bea58a5-bdac-4545-9aee-66961d2bd86b-kube-api-access-gr48c\") pod \"maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm\" (UID: \"2bea58a5-bdac-4545-9aee-66961d2bd86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm"
Oct 30 04:44:24.303586 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:24.303554    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bea58a5-bdac-4545-9aee-66961d2bd86b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm\" (UID: \"2bea58a5-bdac-4545-9aee-66961d2bd86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm"
Oct 30 04:44:24.305922 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:24.305895    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bea58a5-bdac-4545-9aee-66961d2bd86b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm\" (UID: \"2bea58a5-bdac-4545-9aee-66961d2bd86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm"
Oct 30 04:44:24.602625 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:24.602235    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm"
Oct 30 04:44:24.925615 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:24.925541    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm"]
Oct 30 04:44:24.929384 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:44:24.929356    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bea58a5_bdac_4545_9aee_66961d2bd86b.slice/crio-9d04545a4456ce87c3045645cfaf9bada668a64a875bc8171c505795aa5b825c WatchSource:0}: Error finding container 9d04545a4456ce87c3045645cfaf9bada668a64a875bc8171c505795aa5b825c: Status 404 returned error can't find the container with id 9d04545a4456ce87c3045645cfaf9bada668a64a875bc8171c505795aa5b825c
Oct 30 04:44:25.326117 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:25.326084    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm" event={"ID":"2bea58a5-bdac-4545-9aee-66961d2bd86b","Type":"ContainerStarted","Data":"9d04545a4456ce87c3045645cfaf9bada668a64a875bc8171c505795aa5b825c"}
Oct 30 04:44:25.630325 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:44:25.629193    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:44:26.333458 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:26.333426    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm" event={"ID":"2bea58a5-bdac-4545-9aee-66961d2bd86b","Type":"ContainerStarted","Data":"de154c6a5a5a25159626453a9711f9d708e4e4735014da3372091d6d4cc8e781"}
Oct 30 04:44:26.333953 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:26.333549    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm"
Oct 30 04:44:26.375338 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:26.375291    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm" podStartSLOduration=3.00558367 podStartE2EDuration="3.375276832s" totalImagesPullingTime="369.693162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:26.373464507 +0000 UTC m=+2785.785860080" watchObservedRunningTime="2026-10-30 04:44:26.375276832 +0000 UTC m=+2785.787672401"
Oct 30 04:44:32.345003 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:32.344975    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm"
Oct 30 04:44:54.691852 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:54.691805    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"]
Oct 30 04:44:54.692370 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:54.692200    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq" podUID="e31f5bf2-ef53-40df-a264-32d5bfb39dff" containerName="istio-proxy" containerID="cri-o://479c26424bbe4b7ad25fe4aa037ba902aa834bc69029ae909e138e642e77d328" gracePeriod=30
Oct 30 04:44:55.224988 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:55.224955    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"]
Oct 30 04:44:55.618115 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:44:55.618025    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:44:57.157840 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.157806    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"]
Oct 30 04:44:57.162571 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.162542    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.181268 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.181224    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"]
Oct 30 04:44:57.191189 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.191163    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-05bee8-openshift-default-dockercfg-m2glx\""
Oct 30 04:44:57.327959 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.327925    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-envoy\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.328152 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.328052    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-data\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.328152 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.328126    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istiod-ca-cert\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.328312 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.328157    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-workload-certs\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.328312 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.328266    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rckm2\" (UniqueName: \"kubernetes.io/projected/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-kube-api-access-rckm2\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.328418 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.328375    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-credential-socket\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.328527 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.328504    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-podinfo\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.328579 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.328565    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-workload-socket\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.328614 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.328586    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-token\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.429681 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.429586    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rckm2\" (UniqueName: \"kubernetes.io/projected/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-kube-api-access-rckm2\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.429848 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.429705    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-credential-socket\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.429848 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.429826    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-podinfo\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.429983 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.429897    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-workload-socket\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.429983 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.429931    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-token\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.429983 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.429966    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-envoy\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.430146 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.430011    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-data\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.430146 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.430054    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istiod-ca-cert\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.430146 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.430089    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-workload-certs\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.430146 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.430115    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-credential-socket\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.430767 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.430742    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-data\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.431018 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.430998    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-workload-certs\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.431098 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.430999    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-workload-socket\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.431447 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.431425    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istiod-ca-cert\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.432613 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.432586    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-envoy\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.434354 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.433617    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-podinfo\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.453275 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.453200    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-token\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.453622 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.453602    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rckm2\" (UniqueName: \"kubernetes.io/projected/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-kube-api-access-rckm2\") pod \"e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") " pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.477606 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.477582    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:57.840185 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.840093    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"]
Oct 30 04:44:57.843280 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:44:57.843251    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ec0ec0_354e_41f9_897e_ecbd08404b2e.slice/crio-dcb431a02afa5c420820c942d20543f51530b2dbae8ab5bc7a6d865761db1b23 WatchSource:0}: Error finding container dcb431a02afa5c420820c942d20543f51530b2dbae8ab5bc7a6d865761db1b23: Status 404 returned error can't find the container with id dcb431a02afa5c420820c942d20543f51530b2dbae8ab5bc7a6d865761db1b23
Oct 30 04:44:57.846008 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.845970    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:44:57.846124 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.846052    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:44:57.846124 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:57.846097    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:44:58.570825 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:58.570787    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw" event={"ID":"c4ec0ec0-354e-41f9-897e-ecbd08404b2e","Type":"ContainerStarted","Data":"67be90f95774f534380bc661b2248bd32e3065be0904800b5eaef4e8f65ac8c9"}
Oct 30 04:44:58.571342 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:58.570843    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw" event={"ID":"c4ec0ec0-354e-41f9-897e-ecbd08404b2e","Type":"ContainerStarted","Data":"dcb431a02afa5c420820c942d20543f51530b2dbae8ab5bc7a6d865761db1b23"}
Oct 30 04:44:58.637541 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:58.637481    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw" podStartSLOduration=1.637464179 podStartE2EDuration="1.637464179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:58.624077821 +0000 UTC m=+2818.036473395" watchObservedRunningTime="2026-10-30 04:44:58.637464179 +0000 UTC m=+2818.049859753"
Oct 30 04:44:59.478517 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:59.478437    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:59.498741 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:59.498711    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:59.580027 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:59.579971    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:59.581792 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:59.581766    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:44:59.975640 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:44:59.975604    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:45:00.052535 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.052505    4874 patch_prober.go:29] interesting pod/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": read tcp 10.134.0.2:60262->10.134.0.139:15021: read: connection reset by peer" start-of-body=
Oct 30 04:45:00.052707 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.052576    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb" podUID="06eecd27-740b-48d7-903c-182db0dae7ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": read tcp 10.134.0.2:60262->10.134.0.139:15021: read: connection reset by peer"
Oct 30 04:45:00.092338 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.092315    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-data\") pod \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") "
Oct 30 04:45:00.092444 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.092384    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-envoy\") pod \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") "
Oct 30 04:45:00.092488 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.092442    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-workload-certs\") pod \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") "
Oct 30 04:45:00.092533 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.092513    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-podinfo\") pod \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") "
Oct 30 04:45:00.092566 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.092554    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istiod-ca-cert\") pod \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") "
Oct 30 04:45:00.092599 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.092557    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-data" pod "e31f5bf2-ef53-40df-a264-32d5bfb39dff" (UID: "e31f5bf2-ef53-40df-a264-32d5bfb39dff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:45:00.092644 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.092629    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-credential-socket\") pod \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") "
Oct 30 04:45:00.092696 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.092662    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-workload-socket\") pod \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") "
Oct 30 04:45:00.092747 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.092727    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e31f5bf2-ef53-40df-a264-32d5bfb39dff-kube-api-access-kdvxw\" (UniqueName: \"kubernetes.io/projected/e31f5bf2-ef53-40df-a264-32d5bfb39dff-kube-api-access-kdvxw\") pod \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") "
Oct 30 04:45:00.092804 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.092753    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-workload-certs" pod "e31f5bf2-ef53-40df-a264-32d5bfb39dff" (UID: "e31f5bf2-ef53-40df-a264-32d5bfb39dff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:45:00.092863 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.092818    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-token\" (UniqueName: \"kubernetes.io/projected/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-token\") pod \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\" (UID: \"e31f5bf2-ef53-40df-a264-32d5bfb39dff\") "
Oct 30 04:45:00.092922 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.092873    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-credential-socket" pod "e31f5bf2-ef53-40df-a264-32d5bfb39dff" (UID: "e31f5bf2-ef53-40df-a264-32d5bfb39dff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:45:00.093006 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.092982    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-workload-socket" pod "e31f5bf2-ef53-40df-a264-32d5bfb39dff" (UID: "e31f5bf2-ef53-40df-a264-32d5bfb39dff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:45:00.093058 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.092988    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istiod-ca-cert" pod "e31f5bf2-ef53-40df-a264-32d5bfb39dff" (UID: "e31f5bf2-ef53-40df-a264-32d5bfb39dff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:45:00.093276 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.093230    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:45:00.093348 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.093281    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:45:00.093348 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.093291    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:45:00.093348 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.093300    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:45:00.093348 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.093310    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:45:00.094964 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.094941    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-podinfo" pod "e31f5bf2-ef53-40df-a264-32d5bfb39dff" (UID: "e31f5bf2-ef53-40df-a264-32d5bfb39dff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:45:00.095060 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.095037    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-envoy" pod "e31f5bf2-ef53-40df-a264-32d5bfb39dff" (UID: "e31f5bf2-ef53-40df-a264-32d5bfb39dff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:45:00.095102 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.095047    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-token" pod "e31f5bf2-ef53-40df-a264-32d5bfb39dff" (UID: "e31f5bf2-ef53-40df-a264-32d5bfb39dff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:45:00.095102 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.095061    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e31f5bf2-ef53-40df-a264-32d5bfb39dff-kube-api-access-kdvxw" pod "e31f5bf2-ef53-40df-a264-32d5bfb39dff" (UID: "e31f5bf2-ef53-40df-a264-32d5bfb39dff"). InnerVolumeSpecName "kube-api-access-kdvxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:45:00.195739 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.195664    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:45:00.195739 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.195694    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:45:00.195739 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.195705    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e31f5bf2-ef53-40df-a264-32d5bfb39dff-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:45:00.195739 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.195718    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdvxw\" (UniqueName: \"kubernetes.io/projected/e31f5bf2-ef53-40df-a264-32d5bfb39dff-kube-api-access-kdvxw\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:45:00.587797 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.587764    4874 generic.go:356] "Generic (PLEG): container finished" podID="e31f5bf2-ef53-40df-a264-32d5bfb39dff" containerID="479c26424bbe4b7ad25fe4aa037ba902aa834bc69029ae909e138e642e77d328" exitCode=0
Oct 30 04:45:00.588211 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.587816    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"
Oct 30 04:45:00.588211 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.587838    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq" event={"ID":"e31f5bf2-ef53-40df-a264-32d5bfb39dff","Type":"ContainerDied","Data":"479c26424bbe4b7ad25fe4aa037ba902aa834bc69029ae909e138e642e77d328"}
Oct 30 04:45:00.588211 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.587913    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq" event={"ID":"e31f5bf2-ef53-40df-a264-32d5bfb39dff","Type":"ContainerDied","Data":"ea4819458818c22e2930a04834aab735ce348b8316a5f13b02a92f74069644d9"}
Oct 30 04:45:00.588211 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.587948    4874 scope.go:122] "RemoveContainer" containerID="479c26424bbe4b7ad25fe4aa037ba902aa834bc69029ae909e138e642e77d328"
Oct 30 04:45:00.589914 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.589892    4874 generic.go:356] "Generic (PLEG): container finished" podID="06eecd27-740b-48d7-903c-182db0dae7ad" containerID="8163ae0c23b5fbf2851e182dbbad87efd76e629599cfc1623582f8c765610220" exitCode=0
Oct 30 04:45:00.590032 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.589969    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb" event={"ID":"06eecd27-740b-48d7-903c-182db0dae7ad","Type":"ContainerDied","Data":"8163ae0c23b5fbf2851e182dbbad87efd76e629599cfc1623582f8c765610220"}
Oct 30 04:45:00.590446 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.590430    4874 scope.go:122] "RemoveContainer" containerID="8163ae0c23b5fbf2851e182dbbad87efd76e629599cfc1623582f8c765610220"
Oct 30 04:45:00.591839 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.591811    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:45:00.592003 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.591882    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:45:00.592003 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.591916    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:45:00.598575 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.598556    4874 scope.go:122] "RemoveContainer" containerID="479c26424bbe4b7ad25fe4aa037ba902aa834bc69029ae909e138e642e77d328"
Oct 30 04:45:00.598815 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:45:00.598796    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"479c26424bbe4b7ad25fe4aa037ba902aa834bc69029ae909e138e642e77d328\": container with ID starting with 479c26424bbe4b7ad25fe4aa037ba902aa834bc69029ae909e138e642e77d328 not found: ID does not exist" containerID="479c26424bbe4b7ad25fe4aa037ba902aa834bc69029ae909e138e642e77d328"
Oct 30 04:45:00.598901 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.598824    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"479c26424bbe4b7ad25fe4aa037ba902aa834bc69029ae909e138e642e77d328"} err="failed to get container status \"479c26424bbe4b7ad25fe4aa037ba902aa834bc69029ae909e138e642e77d328\": rpc error: code = NotFound desc = could not find container \"479c26424bbe4b7ad25fe4aa037ba902aa834bc69029ae909e138e642e77d328\": container with ID starting with 479c26424bbe4b7ad25fe4aa037ba902aa834bc69029ae909e138e642e77d328 not found: ID does not exist"
Oct 30 04:45:00.690978 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.687318    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"]
Oct 30 04:45:00.691689 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.691663    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-59rmq"]
Oct 30 04:45:00.735369 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:00.735333    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e31f5bf2-ef53-40df-a264-32d5bfb39dff" path="/var/lib/kubelet/pods/e31f5bf2-ef53-40df-a264-32d5bfb39dff/volumes"
Oct 30 04:45:01.606269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:01.603467    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb" event={"ID":"06eecd27-740b-48d7-903c-182db0dae7ad","Type":"ContainerStarted","Data":"0e71fb7f3a0f1f044214c99ced0c32571bc809c6f406d9bc7eadeaeee199e806"}
Oct 30 04:45:01.758310 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:01.746983    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"]
Oct 30 04:45:01.758310 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:01.748013    4874 cpu_manager.go:414] "containerMap: removing container" podUID="e31f5bf2-ef53-40df-a264-32d5bfb39dff" containerName="istio-proxy"
Oct 30 04:45:01.758310 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:01.748034    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e31f5bf2-ef53-40df-a264-32d5bfb39dff" containerName="istio-proxy"
Oct 30 04:45:01.758310 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:01.748345    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="e31f5bf2-ef53-40df-a264-32d5bfb39dff" containerName="istio-proxy"
Oct 30 04:45:01.803541 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:01.803451    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"]
Oct 30 04:45:01.803726 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:01.803575    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:01.889732 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:01.889622    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-c0e964-openshift-default-dockercfg-k2x68\""
Oct 30 04:45:01.922599 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:01.922570    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mlhl\" (UniqueName: \"kubernetes.io/projected/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-kube-api-access-9mlhl\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:01.922778 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:01.922638    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-credential-socket\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:01.922778 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:01.922717    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-podinfo\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:01.922778 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:01.922745    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-token\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:01.922966 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:01.922841    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-workload-certs\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:01.922966 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:01.922925    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-envoy\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:01.923044 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:01.922983    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-data\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:01.923044 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:01.923006    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-workload-socket\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:01.923148 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:01.923043    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istiod-ca-cert\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:02.017308 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.017279    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:45:02.017454 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.017391    4874 patch_prober.go:29] interesting pod/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 04:45:02.017516 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.017447    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb" podUID="06eecd27-740b-48d7-903c-182db0dae7ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 04:45:02.024178 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.024146    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-envoy\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:02.024623 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.024604    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-data\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:02.025158 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.025132    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-workload-socket\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:02.029363 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.025554    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-workload-socket\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:02.029488 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.025072    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-data\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:02.029647 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.029631    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istiod-ca-cert\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:02.037382 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.037355    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mlhl\" (UniqueName: \"kubernetes.io/projected/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-kube-api-access-9mlhl\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:02.037632 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.037615    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-credential-socket\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:02.037759 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.037744    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-podinfo\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:02.037859 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.037846    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-token\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:02.045264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.037996    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-workload-certs\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:02.045517 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.044976    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-credential-socket\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:02.045639 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.030369    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istiod-ca-cert\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:02.046198 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.046173    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-workload-certs\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:02.056939 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.056813    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-envoy\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:02.057656 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.057630    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-podinfo\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:02.093396 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.093229    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-token\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:02.094869 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.094842    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mlhl\" (UniqueName: \"kubernetes.io/projected/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-kube-api-access-9mlhl\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:02.146098 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.146066    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:02.494371 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.494339    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"]
Oct 30 04:45:02.494682 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:45:02.494658    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a79cd0_bb0f_40a4_bb6d_5099288a42e8.slice/crio-3077718e1e612449b618debe995c9856305078a0f1eb23d42b43b272f32b69d8 WatchSource:0}: Error finding container 3077718e1e612449b618debe995c9856305078a0f1eb23d42b43b272f32b69d8: Status 404 returned error can't find the container with id 3077718e1e612449b618debe995c9856305078a0f1eb23d42b43b272f32b69d8
Oct 30 04:45:02.497850 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.497818    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:45:02.497945 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.497884    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:45:02.497945 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.497912    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:45:02.614296 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.614255    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4" event={"ID":"90a79cd0-bb0f-40a4-bb6d-5099288a42e8","Type":"ContainerStarted","Data":"07ee46e07e2c4e12d1eb627dd0bb43c0673052e9861651dc00899a273a6e5e07"}
Oct 30 04:45:02.614296 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.614303    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4" event={"ID":"90a79cd0-bb0f-40a4-bb6d-5099288a42e8","Type":"ContainerStarted","Data":"3077718e1e612449b618debe995c9856305078a0f1eb23d42b43b272f32b69d8"}
Oct 30 04:45:02.692828 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:02.692759    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4" podStartSLOduration=1.69274067 podStartE2EDuration="1.69274067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:02.691608696 +0000 UTC m=+2822.104004270" watchObservedRunningTime="2026-10-30 04:45:02.69274067 +0000 UTC m=+2822.105136244"
Oct 30 04:45:03.021337 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:03.021268    4874 patch_prober.go:29] interesting pod/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 04:45:03.021337 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:03.021330    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb" podUID="06eecd27-740b-48d7-903c-182db0dae7ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 04:45:03.146922 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:03.146890    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:03.152042 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:03.152016    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:03.619035 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:03.618998    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:03.620027 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:03.620000    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:45:04.017546 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:04.017513    4874 patch_prober.go:29] interesting pod/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 04:45:04.017723 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:04.017564    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb" podUID="06eecd27-740b-48d7-903c-182db0dae7ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 04:45:05.024264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:05.020955    4874 patch_prober.go:29] interesting pod/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 04:45:05.024264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:05.021018    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb" podUID="06eecd27-740b-48d7-903c-182db0dae7ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 04:45:05.310277 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:05.310156    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:45:06.017157 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:06.017120    4874 patch_prober.go:29] interesting pod/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 04:45:06.017385 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:06.017175    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb" podUID="06eecd27-740b-48d7-903c-182db0dae7ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 04:45:07.017911 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:07.017873    4874 patch_prober.go:29] interesting pod/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 04:45:07.018461 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:07.017934    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb" podUID="06eecd27-740b-48d7-903c-182db0dae7ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 04:45:08.022835 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:08.022804    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:45:08.023277 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:08.023255    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:45:08.023956 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:08.023933    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 04:45:10.245760 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.245719    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz" podUID="5e850259-0899-4e15-93eb-d3f861d16b81" containerName="main" containerID="cri-o://dd9bf5f144d2f42e01cc9a3bd5e7baf05b5764e27e3688cf9263bc0d217faedc" gracePeriod=45
Oct 30 04:45:10.249982 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.249952    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz" podUID="5e850259-0899-4e15-93eb-d3f861d16b81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.134:8000/ready\": EOF"
Oct 30 04:45:10.522457 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.522433    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:45:10.629947 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.629912    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-home\" (UniqueName: \"kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-home\") pod \"5e850259-0899-4e15-93eb-d3f861d16b81\" (UID: \"5e850259-0899-4e15-93eb-d3f861d16b81\") "
Oct 30 04:45:10.630114 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.630011    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-tmp-dir\") pod \"5e850259-0899-4e15-93eb-d3f861d16b81\" (UID: \"5e850259-0899-4e15-93eb-d3f861d16b81\") "
Oct 30 04:45:10.630114 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.630067    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-model-cache\") pod \"5e850259-0899-4e15-93eb-d3f861d16b81\" (UID: \"5e850259-0899-4e15-93eb-d3f861d16b81\") "
Oct 30 04:45:10.630301 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.630119    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e850259-0899-4e15-93eb-d3f861d16b81-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e850259-0899-4e15-93eb-d3f861d16b81-tls-certs\") pod \"5e850259-0899-4e15-93eb-d3f861d16b81\" (UID: \"5e850259-0899-4e15-93eb-d3f861d16b81\") "
Oct 30 04:45:10.630301 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.630147    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e850259-0899-4e15-93eb-d3f861d16b81-kube-api-access-64sqb\" (UniqueName: \"kubernetes.io/projected/5e850259-0899-4e15-93eb-d3f861d16b81-kube-api-access-64sqb\") pod \"5e850259-0899-4e15-93eb-d3f861d16b81\" (UID: \"5e850259-0899-4e15-93eb-d3f861d16b81\") "
Oct 30 04:45:10.630301 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.630178    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-dshm\") pod \"5e850259-0899-4e15-93eb-d3f861d16b81\" (UID: \"5e850259-0899-4e15-93eb-d3f861d16b81\") "
Oct 30 04:45:10.630301 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.630262    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-home" pod "5e850259-0899-4e15-93eb-d3f861d16b81" (UID: "5e850259-0899-4e15-93eb-d3f861d16b81"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:45:10.630301 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.630269    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-tmp-dir" pod "5e850259-0899-4e15-93eb-d3f861d16b81" (UID: "5e850259-0899-4e15-93eb-d3f861d16b81"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:45:10.630549 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.630418    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-model-cache" pod "5e850259-0899-4e15-93eb-d3f861d16b81" (UID: "5e850259-0899-4e15-93eb-d3f861d16b81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:45:10.630808 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.630784    4874 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-tmp-dir\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:45:10.630808 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.630811    4874 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-model-cache\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:45:10.631005 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.630824    4874 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-home\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:45:10.632596 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.632536    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e850259-0899-4e15-93eb-d3f861d16b81-kube-api-access-64sqb" pod "5e850259-0899-4e15-93eb-d3f861d16b81" (UID: "5e850259-0899-4e15-93eb-d3f861d16b81"). InnerVolumeSpecName "kube-api-access-64sqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:45:10.632721 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.632694    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-dshm" pod "5e850259-0899-4e15-93eb-d3f861d16b81" (UID: "5e850259-0899-4e15-93eb-d3f861d16b81"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:45:10.632809 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.632789    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e850259-0899-4e15-93eb-d3f861d16b81-tls-certs" pod "5e850259-0899-4e15-93eb-d3f861d16b81" (UID: "5e850259-0899-4e15-93eb-d3f861d16b81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:45:10.672732 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.672706    4874 generic.go:356] "Generic (PLEG): container finished" podID="5e850259-0899-4e15-93eb-d3f861d16b81" containerID="dd9bf5f144d2f42e01cc9a3bd5e7baf05b5764e27e3688cf9263bc0d217faedc" exitCode=0
Oct 30 04:45:10.672840 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.672754    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"
Oct 30 04:45:10.672840 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.672797    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz" event={"ID":"5e850259-0899-4e15-93eb-d3f861d16b81","Type":"ContainerDied","Data":"dd9bf5f144d2f42e01cc9a3bd5e7baf05b5764e27e3688cf9263bc0d217faedc"}
Oct 30 04:45:10.672927 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.672847    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz" event={"ID":"5e850259-0899-4e15-93eb-d3f861d16b81","Type":"ContainerDied","Data":"af23a1373478865893fc05ca01c48b8dd289fb00a67e426330602cd16944129e"}
Oct 30 04:45:10.672927 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.672868    4874 scope.go:122] "RemoveContainer" containerID="dd9bf5f144d2f42e01cc9a3bd5e7baf05b5764e27e3688cf9263bc0d217faedc"
Oct 30 04:45:10.682651 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.682635    4874 scope.go:122] "RemoveContainer" containerID="dd9bf5f144d2f42e01cc9a3bd5e7baf05b5764e27e3688cf9263bc0d217faedc"
Oct 30 04:45:10.682900 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:45:10.682882    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd9bf5f144d2f42e01cc9a3bd5e7baf05b5764e27e3688cf9263bc0d217faedc\": container with ID starting with dd9bf5f144d2f42e01cc9a3bd5e7baf05b5764e27e3688cf9263bc0d217faedc not found: ID does not exist" containerID="dd9bf5f144d2f42e01cc9a3bd5e7baf05b5764e27e3688cf9263bc0d217faedc"
Oct 30 04:45:10.682947 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.682908    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd9bf5f144d2f42e01cc9a3bd5e7baf05b5764e27e3688cf9263bc0d217faedc"} err="failed to get container status \"dd9bf5f144d2f42e01cc9a3bd5e7baf05b5764e27e3688cf9263bc0d217faedc\": rpc error: code = NotFound desc = could not find container \"dd9bf5f144d2f42e01cc9a3bd5e7baf05b5764e27e3688cf9263bc0d217faedc\": container with ID starting with dd9bf5f144d2f42e01cc9a3bd5e7baf05b5764e27e3688cf9263bc0d217faedc not found: ID does not exist"
Oct 30 04:45:10.732450 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.732421    4874 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e850259-0899-4e15-93eb-d3f861d16b81-tls-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:45:10.732599 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.732455    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64sqb\" (UniqueName: \"kubernetes.io/projected/5e850259-0899-4e15-93eb-d3f861d16b81-kube-api-access-64sqb\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:45:10.732599 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.732470    4874 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e850259-0899-4e15-93eb-d3f861d16b81-dshm\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:45:10.737569 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.737543    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"]
Oct 30 04:45:10.737668 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:10.737581    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-495ebe/auth-test-model-495ebe-kserve-6b76d48fd5-bbdqz"]
Oct 30 04:45:12.729259 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:12.729210    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e850259-0899-4e15-93eb-d3f861d16b81" path="/var/lib/kubelet/pods/5e850259-0899-4e15-93eb-d3f861d16b81/volumes"
Oct 30 04:45:22.562566 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:22.562489    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h"]
Oct 30 04:45:22.563018 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:22.563005    4874 cpu_manager.go:414] "containerMap: removing container" podUID="5e850259-0899-4e15-93eb-d3f861d16b81" containerName="main"
Oct 30 04:45:22.563067 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:22.563019    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e850259-0899-4e15-93eb-d3f861d16b81" containerName="main"
Oct 30 04:45:22.563126 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:22.563117    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e850259-0899-4e15-93eb-d3f861d16b81" containerName="main"
Oct 30 04:45:22.566378 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:22.566357    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h"
Oct 30 04:45:22.580694 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:22.580673    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-6378cc-serving-cert\""
Oct 30 04:45:22.596254 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:22.596216    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h"]
Oct 30 04:45:22.658344 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:22.658315    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70bb7edd-3719-4c86-b358-2039dbd00675-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h\" (UID: \"70bb7edd-3719-4c86-b358-2039dbd00675\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h"
Oct 30 04:45:22.658487 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:22.658377    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70bb7edd-3719-4c86-b358-2039dbd00675-metrics-certs\") pod \"maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h\" (UID: \"70bb7edd-3719-4c86-b358-2039dbd00675\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h"
Oct 30 04:45:22.658531 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:22.658492    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70bb7edd-3719-4c86-b358-2039dbd00675-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h\" (UID: \"70bb7edd-3719-4c86-b358-2039dbd00675\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h"
Oct 30 04:45:22.658566 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:22.658548    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsxtj\" (UniqueName: \"kubernetes.io/projected/70bb7edd-3719-4c86-b358-2039dbd00675-kube-api-access-wsxtj\") pod \"maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h\" (UID: \"70bb7edd-3719-4c86-b358-2039dbd00675\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h"
Oct 30 04:45:22.778229 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:22.771212    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70bb7edd-3719-4c86-b358-2039dbd00675-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h\" (UID: \"70bb7edd-3719-4c86-b358-2039dbd00675\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h"
Oct 30 04:45:22.778229 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:22.771339    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsxtj\" (UniqueName: \"kubernetes.io/projected/70bb7edd-3719-4c86-b358-2039dbd00675-kube-api-access-wsxtj\") pod \"maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h\" (UID: \"70bb7edd-3719-4c86-b358-2039dbd00675\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h"
Oct 30 04:45:22.778229 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:22.771474    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70bb7edd-3719-4c86-b358-2039dbd00675-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h\" (UID: \"70bb7edd-3719-4c86-b358-2039dbd00675\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h"
Oct 30 04:45:22.778229 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:22.771558    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70bb7edd-3719-4c86-b358-2039dbd00675-metrics-certs\") pod \"maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h\" (UID: \"70bb7edd-3719-4c86-b358-2039dbd00675\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h"
Oct 30 04:45:22.779364 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:45:22.779338    4874 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-serving-cert: secret "maas-api-e2e-extmod-a-6378cc-serving-cert" not found
Oct 30 04:45:22.789307 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:45:22.779575    4874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70bb7edd-3719-4c86-b358-2039dbd00675-maas-api-tls podName:70bb7edd-3719-4c86-b358-2039dbd00675 nodeName:}" failed. No retries permitted until 2026-10-30 04:45:23.2795507 +0000 UTC m=+2842.691946263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/70bb7edd-3719-4c86-b358-2039dbd00675-maas-api-tls") pod "maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h" (UID: "70bb7edd-3719-4c86-b358-2039dbd00675") : secret "maas-api-e2e-extmod-a-6378cc-serving-cert" not found
Oct 30 04:45:22.789307 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:22.785169    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70bb7edd-3719-4c86-b358-2039dbd00675-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h\" (UID: \"70bb7edd-3719-4c86-b358-2039dbd00675\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h"
Oct 30 04:45:22.789307 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:22.787741    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70bb7edd-3719-4c86-b358-2039dbd00675-metrics-certs\") pod \"maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h\" (UID: \"70bb7edd-3719-4c86-b358-2039dbd00675\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h"
Oct 30 04:45:22.803654 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:22.803619    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsxtj\" (UniqueName: \"kubernetes.io/projected/70bb7edd-3719-4c86-b358-2039dbd00675-kube-api-access-wsxtj\") pod \"maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h\" (UID: \"70bb7edd-3719-4c86-b358-2039dbd00675\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h"
Oct 30 04:45:23.379383 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:23.379348    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70bb7edd-3719-4c86-b358-2039dbd00675-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h\" (UID: \"70bb7edd-3719-4c86-b358-2039dbd00675\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h"
Oct 30 04:45:23.381729 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:23.381698    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70bb7edd-3719-4c86-b358-2039dbd00675-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h\" (UID: \"70bb7edd-3719-4c86-b358-2039dbd00675\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h"
Oct 30 04:45:23.479299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:23.479269    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h"
Oct 30 04:45:23.789160 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:23.789136    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h"]
Oct 30 04:45:23.791591 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:45:23.791563    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70bb7edd_3719_4c86_b358_2039dbd00675.slice/crio-f5158b8dcfe9380dd48e1db745ba9bbff70b8ca36489d10434b104faf116fcde WatchSource:0}: Error finding container f5158b8dcfe9380dd48e1db745ba9bbff70b8ca36489d10434b104faf116fcde: Status 404 returned error can't find the container with id f5158b8dcfe9380dd48e1db745ba9bbff70b8ca36489d10434b104faf116fcde
Oct 30 04:45:23.792927 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:23.792907    4874 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 04:45:24.760412 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:24.760376    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h" event={"ID":"70bb7edd-3719-4c86-b358-2039dbd00675","Type":"ContainerStarted","Data":"2e88f8afd6aab7da47be79c82173743943faa1c7a09921870bbe3c8a37de0394"}
Oct 30 04:45:24.760596 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:24.760549    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h"
Oct 30 04:45:24.760654 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:24.760594    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h" event={"ID":"70bb7edd-3719-4c86-b358-2039dbd00675","Type":"ContainerStarted","Data":"f5158b8dcfe9380dd48e1db745ba9bbff70b8ca36489d10434b104faf116fcde"}
Oct 30 04:45:24.803461 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:24.803412    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h" podStartSLOduration=2.4378397720000002 podStartE2EDuration="2.803399523s" totalImagesPullingTime="365.559751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:24.801525485 +0000 UTC m=+2844.213921057" watchObservedRunningTime="2026-10-30 04:45:24.803399523 +0000 UTC m=+2844.215795162"
Oct 30 04:45:25.440963 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.440929    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"]
Oct 30 04:45:25.443959 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.443941    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.457801 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.457783    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-247158-openshift-default-dockercfg-rvw8n\""
Oct 30 04:45:25.472529 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.472506    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"]
Oct 30 04:45:25.610098 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:45:25.610061    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:45:25.611673 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.611652    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-token\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.611746 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.611684    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-data\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.611816 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.611798    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwrpb\" (UniqueName: \"kubernetes.io/projected/dd40b6ca-451d-4750-a621-920fa4d7425d-kube-api-access-dwrpb\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.611863 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.611842    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-workload-socket\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.611938 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.611926    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-envoy\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.611986 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.611974    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-podinfo\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.612045 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.612030    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-credential-socket\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.612083 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.612067    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-workload-certs\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.612129 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.612094    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd40b6ca-451d-4750-a621-920fa4d7425d-istiod-ca-cert\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.712960 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.712886    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd40b6ca-451d-4750-a621-920fa4d7425d-istiod-ca-cert\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.712960 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.712945    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-token\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.713153 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.712973    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-data\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.713153 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.713033    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwrpb\" (UniqueName: \"kubernetes.io/projected/dd40b6ca-451d-4750-a621-920fa4d7425d-kube-api-access-dwrpb\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.713153 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.713078    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-workload-socket\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.713343 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.713157    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-envoy\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.713343 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.713218    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-podinfo\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.713343 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.713259    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-credential-socket\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.713343 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.713304    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-workload-certs\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.713537 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.713439    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-data\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.713638 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.713593    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-workload-socket\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.713761 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.713695    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd40b6ca-451d-4750-a621-920fa4d7425d-istiod-ca-cert\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.713813 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.713798    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-workload-certs\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.713877 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.713854    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-credential-socket\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.715581 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.715555    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-envoy\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.715676 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.715661    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-podinfo\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.735313 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.735290    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-token\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.735392 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.735324    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwrpb\" (UniqueName: \"kubernetes.io/projected/dd40b6ca-451d-4750-a621-920fa4d7425d-kube-api-access-dwrpb\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:25.756402 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:25.756381    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:26.079233 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:26.079201    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"]
Oct 30 04:45:26.082353 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:45:26.082323    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd40b6ca_451d_4750_a621_920fa4d7425d.slice/crio-78b86bfc69f18869406e7414849d9bd9b531401f5e4cc33224b8d02bf6e98a9d WatchSource:0}: Error finding container 78b86bfc69f18869406e7414849d9bd9b531401f5e4cc33224b8d02bf6e98a9d: Status 404 returned error can't find the container with id 78b86bfc69f18869406e7414849d9bd9b531401f5e4cc33224b8d02bf6e98a9d
Oct 30 04:45:26.085181 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:26.085150    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:45:26.085281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:26.085213    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:45:26.085281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:26.085260    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:45:26.775033 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:26.774992    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn" event={"ID":"dd40b6ca-451d-4750-a621-920fa4d7425d","Type":"ContainerStarted","Data":"140e2cce000819ab5f85f856edd6f82b57c86dba05951825d982caab1bb2b8d5"}
Oct 30 04:45:26.775204 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:26.775049    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn" event={"ID":"dd40b6ca-451d-4750-a621-920fa4d7425d","Type":"ContainerStarted","Data":"78b86bfc69f18869406e7414849d9bd9b531401f5e4cc33224b8d02bf6e98a9d"}
Oct 30 04:45:26.842161 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:26.842103    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn" podStartSLOduration=1.84208891 podStartE2EDuration="1.84208891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:26.829322211 +0000 UTC m=+2846.241717803" watchObservedRunningTime="2026-10-30 04:45:26.84208891 +0000 UTC m=+2846.254484485"
Oct 30 04:45:27.757449 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:27.757416    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:27.759075 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:27.759052    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 30 04:45:27.759160 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:27.759103    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn" podUID="dd40b6ca-451d-4750-a621-920fa4d7425d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 30 04:45:28.757310 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:28.757279    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 30 04:45:28.757509 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:28.757334    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn" podUID="dd40b6ca-451d-4750-a621-920fa4d7425d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 30 04:45:29.756804 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:29.756775    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 30 04:45:29.756956 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:29.756824    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn" podUID="dd40b6ca-451d-4750-a621-920fa4d7425d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 30 04:45:30.757326 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:30.757295    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 30 04:45:30.757816 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:30.757350    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn" podUID="dd40b6ca-451d-4750-a621-920fa4d7425d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 30 04:45:30.792734 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:30.792706    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h"
Oct 30 04:45:31.757444 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:31.757410    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 30 04:45:31.757842 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:31.757464    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn" podUID="dd40b6ca-451d-4750-a621-920fa4d7425d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 30 04:45:33.068012 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:33.067980    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:33.068795 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:33.068772    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:33.069571 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:33.069548    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 04:45:36.463862 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:36.463826    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4"]
Oct 30 04:45:36.466882 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:36.466861    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4"
Oct 30 04:45:36.485139 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:36.485115    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-c0e964-serving-cert\""
Oct 30 04:45:36.486855 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:36.486830    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/142fff76-0542-40d8-bea6-f46a50366b1d-metrics-certs\") pod \"maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4\" (UID: \"142fff76-0542-40d8-bea6-f46a50366b1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4"
Oct 30 04:45:36.486963 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:36.486902    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/142fff76-0542-40d8-bea6-f46a50366b1d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4\" (UID: \"142fff76-0542-40d8-bea6-f46a50366b1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4"
Oct 30 04:45:36.487041 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:36.487016    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/142fff76-0542-40d8-bea6-f46a50366b1d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4\" (UID: \"142fff76-0542-40d8-bea6-f46a50366b1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4"
Oct 30 04:45:36.487147 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:36.487123    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vdj7\" (UniqueName: \"kubernetes.io/projected/142fff76-0542-40d8-bea6-f46a50366b1d-kube-api-access-6vdj7\") pod \"maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4\" (UID: \"142fff76-0542-40d8-bea6-f46a50366b1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4"
Oct 30 04:45:36.498037 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:36.498013    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4"]
Oct 30 04:45:36.588494 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:36.588460    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/142fff76-0542-40d8-bea6-f46a50366b1d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4\" (UID: \"142fff76-0542-40d8-bea6-f46a50366b1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4"
Oct 30 04:45:36.588686 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:36.588524    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/142fff76-0542-40d8-bea6-f46a50366b1d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4\" (UID: \"142fff76-0542-40d8-bea6-f46a50366b1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4"
Oct 30 04:45:36.588686 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:36.588560    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vdj7\" (UniqueName: \"kubernetes.io/projected/142fff76-0542-40d8-bea6-f46a50366b1d-kube-api-access-6vdj7\") pod \"maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4\" (UID: \"142fff76-0542-40d8-bea6-f46a50366b1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4"
Oct 30 04:45:36.588686 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:36.588659    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/142fff76-0542-40d8-bea6-f46a50366b1d-metrics-certs\") pod \"maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4\" (UID: \"142fff76-0542-40d8-bea6-f46a50366b1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4"
Oct 30 04:45:36.589355 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:36.589333    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/142fff76-0542-40d8-bea6-f46a50366b1d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4\" (UID: \"142fff76-0542-40d8-bea6-f46a50366b1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4"
Oct 30 04:45:36.590933 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:36.590911    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/142fff76-0542-40d8-bea6-f46a50366b1d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4\" (UID: \"142fff76-0542-40d8-bea6-f46a50366b1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4"
Oct 30 04:45:36.591107 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:36.591084    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/142fff76-0542-40d8-bea6-f46a50366b1d-metrics-certs\") pod \"maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4\" (UID: \"142fff76-0542-40d8-bea6-f46a50366b1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4"
Oct 30 04:45:36.613295 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:36.613273    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vdj7\" (UniqueName: \"kubernetes.io/projected/142fff76-0542-40d8-bea6-f46a50366b1d-kube-api-access-6vdj7\") pod \"maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4\" (UID: \"142fff76-0542-40d8-bea6-f46a50366b1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4"
Oct 30 04:45:36.779392 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:36.779359    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4"
Oct 30 04:45:37.082927 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:37.082901    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4"]
Oct 30 04:45:37.085048 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:45:37.085017    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod142fff76_0542_40d8_bea6_f46a50366b1d.slice/crio-ec09339352a110e83b50387f7d9aa72b9f6afbb194996f1c927ee45da6deadf5 WatchSource:0}: Error finding container ec09339352a110e83b50387f7d9aa72b9f6afbb194996f1c927ee45da6deadf5: Status 404 returned error can't find the container with id ec09339352a110e83b50387f7d9aa72b9f6afbb194996f1c927ee45da6deadf5
Oct 30 04:45:37.859541 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:37.859507    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4" event={"ID":"142fff76-0542-40d8-bea6-f46a50366b1d","Type":"ContainerStarted","Data":"5b341ddf58e43603995f97fbfc2996b6f629a00c846c5ae0266698043f6fdf46"}
Oct 30 04:45:37.859921 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:37.859565    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4" event={"ID":"142fff76-0542-40d8-bea6-f46a50366b1d","Type":"ContainerStarted","Data":"ec09339352a110e83b50387f7d9aa72b9f6afbb194996f1c927ee45da6deadf5"}
Oct 30 04:45:37.859921 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:37.859593    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4"
Oct 30 04:45:37.912622 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:37.910524    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4" podStartSLOduration=1.5806735509999998 podStartE2EDuration="1.910501691s" totalImagesPullingTime="329.82814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:37.900462835 +0000 UTC m=+2857.312858411" watchObservedRunningTime="2026-10-30 04:45:37.910501691 +0000 UTC m=+2857.322897265"
Oct 30 04:45:43.780716 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:43.780683    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:45:43.901788 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:43.901757    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4"
Oct 30 04:45:53.974172 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:53.974078    4874 generic.go:356] "Generic (PLEG): container finished" podID="c4ec0ec0-354e-41f9-897e-ecbd08404b2e" containerID="67be90f95774f534380bc661b2248bd32e3065be0904800b5eaef4e8f65ac8c9" exitCode=0
Oct 30 04:45:53.974172 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:53.974152    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw" event={"ID":"c4ec0ec0-354e-41f9-897e-ecbd08404b2e","Type":"ContainerDied","Data":"67be90f95774f534380bc661b2248bd32e3065be0904800b5eaef4e8f65ac8c9"}
Oct 30 04:45:53.974641 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:53.974558    4874 scope.go:122] "RemoveContainer" containerID="67be90f95774f534380bc661b2248bd32e3065be0904800b5eaef4e8f65ac8c9"
Oct 30 04:45:53.976126 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:53.976101    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:45:53.976178 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:53.976170    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:45:53.976213 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:53.976201    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:45:54.995973 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:54.995929    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw" event={"ID":"c4ec0ec0-354e-41f9-897e-ecbd08404b2e","Type":"ContainerStarted","Data":"00736f477cb56865fbc045a7404d67a6c67ac5bf01df7e910b151a1b8850e8d2"}
Oct 30 04:45:55.479584 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:55.479550    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:45:55.479974 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:55.479950    4874 patch_prober.go:29] interesting pod/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 30 04:45:55.480132 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:55.480109    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw" podUID="c4ec0ec0-354e-41f9-897e-ecbd08404b2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 30 04:45:55.621810 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:45:55.621772    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:45:56.483263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:56.480164    4874 patch_prober.go:29] interesting pod/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 30 04:45:56.483263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:56.480279    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw" podUID="c4ec0ec0-354e-41f9-897e-ecbd08404b2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 30 04:45:57.486268 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:57.485666    4874 patch_prober.go:29] interesting pod/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 30 04:45:57.486268 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:57.485726    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw" podUID="c4ec0ec0-354e-41f9-897e-ecbd08404b2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 30 04:45:58.482625 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:58.482586    4874 patch_prober.go:29] interesting pod/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 30 04:45:58.482929 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:58.482900    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw" podUID="c4ec0ec0-354e-41f9-897e-ecbd08404b2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 30 04:45:59.484582 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:59.484549    4874 patch_prober.go:29] interesting pod/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 30 04:45:59.485147 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:59.485116    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw" podUID="c4ec0ec0-354e-41f9-897e-ecbd08404b2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 30 04:45:59.585263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:45:59.583332    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:46:00.482885 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:00.482853    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:46:00.483806 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:00.483786    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 04:46:09.096644 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.096603    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"]
Oct 30 04:46:09.104334 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.104307    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.125880 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.125851    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-b195d0-openshift-default-dockercfg-gkwn4\""
Oct 30 04:46:09.139463 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.139433    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"]
Oct 30 04:46:09.232711 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.232680    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7abcd90-6f30-44af-bdc4-760820351783-istiod-ca-cert\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.232878 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.232723    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-credential-socket\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.232878 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.232825    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7abcd90-6f30-44af-bdc4-760820351783-istio-podinfo\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.232878 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.232851    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7abcd90-6f30-44af-bdc4-760820351783-istio-token\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.233029 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.233005    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-istio-data\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.233068 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.233041    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-workload-socket\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.233068 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.233063    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-istio-envoy\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.233172 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.233153    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-workload-certs\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.233219 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.233191    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzb9x\" (UniqueName: \"kubernetes.io/projected/c7abcd90-6f30-44af-bdc4-760820351783-kube-api-access-lzb9x\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.334047 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.334015    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-istio-data\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.334264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.334054    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-workload-socket\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.334264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.334082    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-istio-envoy\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.334264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.334128    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-workload-certs\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.334264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.334172    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzb9x\" (UniqueName: \"kubernetes.io/projected/c7abcd90-6f30-44af-bdc4-760820351783-kube-api-access-lzb9x\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.334264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.334259    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7abcd90-6f30-44af-bdc4-760820351783-istiod-ca-cert\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.334628 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.334306    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-credential-socket\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.334628 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.334386    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7abcd90-6f30-44af-bdc4-760820351783-istio-podinfo\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.334628 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.334425    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7abcd90-6f30-44af-bdc4-760820351783-istio-token\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.334628 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.334472    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-workload-socket\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.334628 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.334621    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-workload-certs\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.334930 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.334732    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-credential-socket\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.334930 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.334843    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-istio-data\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.335092 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.335068    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7abcd90-6f30-44af-bdc4-760820351783-istiod-ca-cert\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.336438 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.336418    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-istio-envoy\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.336862 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.336833    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7abcd90-6f30-44af-bdc4-760820351783-istio-podinfo\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.359445 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.359396    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7abcd90-6f30-44af-bdc4-760820351783-istio-token\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.359559 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.359541    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzb9x\" (UniqueName: \"kubernetes.io/projected/c7abcd90-6f30-44af-bdc4-760820351783-kube-api-access-lzb9x\") pod \"e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") " pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.423080 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.423058    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:09.752516 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.752492    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"]
Oct 30 04:46:09.754573 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:46:09.754533    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7abcd90_6f30_44af_bdc4_760820351783.slice/crio-e3ec4c115e9010847ea5d9df56336d053119445eb41cc30c4184388bd6bd8d3a WatchSource:0}: Error finding container e3ec4c115e9010847ea5d9df56336d053119445eb41cc30c4184388bd6bd8d3a: Status 404 returned error can't find the container with id e3ec4c115e9010847ea5d9df56336d053119445eb41cc30c4184388bd6bd8d3a
Oct 30 04:46:09.759827 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.759772    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:46:09.759967 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.759928    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:46:09.760032 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:09.760014    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:46:10.119295 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:10.119186    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l" event={"ID":"c7abcd90-6f30-44af-bdc4-760820351783","Type":"ContainerStarted","Data":"050398fb1f680c4957e138b470dc3a80d22c3535671abfae8715fb6ce1a29536"}
Oct 30 04:46:10.119732 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:10.119311    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l" event={"ID":"c7abcd90-6f30-44af-bdc4-760820351783","Type":"ContainerStarted","Data":"e3ec4c115e9010847ea5d9df56336d053119445eb41cc30c4184388bd6bd8d3a"}
Oct 30 04:46:10.165425 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:10.165378    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l" podStartSLOduration=1.165365248 podStartE2EDuration="1.165365248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:46:10.162347809 +0000 UTC m=+2889.574743382" watchObservedRunningTime="2026-10-30 04:46:10.165365248 +0000 UTC m=+2889.577760821"
Oct 30 04:46:10.422142 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:10.422045    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:46:10.423428 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:10.423405    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:10.491060 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:10.491028    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:11.125023 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:11.124991    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:11.125873 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:11.125852    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:46:19.661405 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:19.661364    4874 patch_prober.go:29] interesting pod/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 04:46:19.661869 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:19.661463    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4" podUID="90a79cd0-bb0f-40a4-bb6d-5099288a42e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 04:46:21.188300 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:21.188268    4874 generic.go:356] "Generic (PLEG): container finished" podID="90a79cd0-bb0f-40a4-bb6d-5099288a42e8" containerID="07ee46e07e2c4e12d1eb627dd0bb43c0673052e9861651dc00899a273a6e5e07" exitCode=0
Oct 30 04:46:21.188803 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:21.188338    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4" event={"ID":"90a79cd0-bb0f-40a4-bb6d-5099288a42e8","Type":"ContainerDied","Data":"07ee46e07e2c4e12d1eb627dd0bb43c0673052e9861651dc00899a273a6e5e07"}
Oct 30 04:46:21.189072 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:21.189045    4874 scope.go:122] "RemoveContainer" containerID="07ee46e07e2c4e12d1eb627dd0bb43c0673052e9861651dc00899a273a6e5e07"
Oct 30 04:46:21.190675 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:21.190645    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:46:21.190763 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:21.190716    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:46:21.190763 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:21.190745    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:46:22.202584 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:22.202550    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4" event={"ID":"90a79cd0-bb0f-40a4-bb6d-5099288a42e8","Type":"ContainerStarted","Data":"78b2a4b3b1484fad04de8b1d0911a6b074331c3049cdf98f47ce918f93cd49ba"}
Oct 30 04:46:23.147047 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:23.147009    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:46:23.147284 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:23.147116    4874 patch_prober.go:29] interesting pod/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 04:46:23.147284 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:23.147176    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4" podUID="90a79cd0-bb0f-40a4-bb6d-5099288a42e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 04:46:24.147023 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:24.146989    4874 patch_prober.go:29] interesting pod/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 04:46:24.147429 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:24.147045    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4" podUID="90a79cd0-bb0f-40a4-bb6d-5099288a42e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 04:46:25.147295 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:25.147259    4874 patch_prober.go:29] interesting pod/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 04:46:25.147691 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:25.147323    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4" podUID="90a79cd0-bb0f-40a4-bb6d-5099288a42e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 04:46:25.617065 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:46:25.617030    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:46:26.147065 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:26.147025    4874 patch_prober.go:29] interesting pod/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 04:46:26.147258 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:26.147081    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4" podUID="90a79cd0-bb0f-40a4-bb6d-5099288a42e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 04:46:27.146813 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:27.146780    4874 patch_prober.go:29] interesting pod/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 04:46:27.147195 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:27.146831    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4" podUID="90a79cd0-bb0f-40a4-bb6d-5099288a42e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 04:46:28.151536 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:28.151503    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:46:28.151928 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:28.151883    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:46:28.152588 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:28.152569    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 04:46:37.285692 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:37.285658    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:46:45.684877 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:45.684846    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn"]
Oct 30 04:46:45.689307 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:45.689290    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn"
Oct 30 04:46:45.725268 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:45.716928    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-b195d0-serving-cert\""
Oct 30 04:46:45.727203 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:45.727178    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn"]
Oct 30 04:46:45.857221 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:45.857189    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20246613-3c91-4fc6-9c96-0e0b81709958-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn\" (UID: \"20246613-3c91-4fc6-9c96-0e0b81709958\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn"
Oct 30 04:46:45.857413 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:45.857320    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20246613-3c91-4fc6-9c96-0e0b81709958-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn\" (UID: \"20246613-3c91-4fc6-9c96-0e0b81709958\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn"
Oct 30 04:46:45.857557 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:45.857531    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20246613-3c91-4fc6-9c96-0e0b81709958-metrics-certs\") pod \"maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn\" (UID: \"20246613-3c91-4fc6-9c96-0e0b81709958\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn"
Oct 30 04:46:45.857616 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:45.857601    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fprvf\" (UniqueName: \"kubernetes.io/projected/20246613-3c91-4fc6-9c96-0e0b81709958-kube-api-access-fprvf\") pod \"maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn\" (UID: \"20246613-3c91-4fc6-9c96-0e0b81709958\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn"
Oct 30 04:46:45.959324 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:45.959206    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20246613-3c91-4fc6-9c96-0e0b81709958-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn\" (UID: \"20246613-3c91-4fc6-9c96-0e0b81709958\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn"
Oct 30 04:46:45.959502 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:45.959336    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20246613-3c91-4fc6-9c96-0e0b81709958-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn\" (UID: \"20246613-3c91-4fc6-9c96-0e0b81709958\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn"
Oct 30 04:46:45.959502 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:45.959362    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20246613-3c91-4fc6-9c96-0e0b81709958-metrics-certs\") pod \"maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn\" (UID: \"20246613-3c91-4fc6-9c96-0e0b81709958\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn"
Oct 30 04:46:45.959502 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:45.959390    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fprvf\" (UniqueName: \"kubernetes.io/projected/20246613-3c91-4fc6-9c96-0e0b81709958-kube-api-access-fprvf\") pod \"maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn\" (UID: \"20246613-3c91-4fc6-9c96-0e0b81709958\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn"
Oct 30 04:46:45.960062 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:45.960036    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20246613-3c91-4fc6-9c96-0e0b81709958-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn\" (UID: \"20246613-3c91-4fc6-9c96-0e0b81709958\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn"
Oct 30 04:46:45.961724 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:45.961704    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20246613-3c91-4fc6-9c96-0e0b81709958-metrics-certs\") pod \"maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn\" (UID: \"20246613-3c91-4fc6-9c96-0e0b81709958\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn"
Oct 30 04:46:45.961825 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:45.961730    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20246613-3c91-4fc6-9c96-0e0b81709958-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn\" (UID: \"20246613-3c91-4fc6-9c96-0e0b81709958\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn"
Oct 30 04:46:45.984182 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:45.984158    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fprvf\" (UniqueName: \"kubernetes.io/projected/20246613-3c91-4fc6-9c96-0e0b81709958-kube-api-access-fprvf\") pod \"maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn\" (UID: \"20246613-3c91-4fc6-9c96-0e0b81709958\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn"
Oct 30 04:46:46.012322 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:46.012286    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn"
Oct 30 04:46:46.322622 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:46.322592    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn"]
Oct 30 04:46:46.324539 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:46:46.324510    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20246613_3c91_4fc6_9c96_0e0b81709958.slice/crio-a89b7a18af55f65afa36b09bed6b90be79386fe4963a10b8f8f4e59a047ab466 WatchSource:0}: Error finding container a89b7a18af55f65afa36b09bed6b90be79386fe4963a10b8f8f4e59a047ab466: Status 404 returned error can't find the container with id a89b7a18af55f65afa36b09bed6b90be79386fe4963a10b8f8f4e59a047ab466
Oct 30 04:46:46.346155 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:46.346128    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn" event={"ID":"20246613-3c91-4fc6-9c96-0e0b81709958","Type":"ContainerStarted","Data":"a89b7a18af55f65afa36b09bed6b90be79386fe4963a10b8f8f4e59a047ab466"}
Oct 30 04:46:47.353779 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:47.353742    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn" event={"ID":"20246613-3c91-4fc6-9c96-0e0b81709958","Type":"ContainerStarted","Data":"15d09438ac875d56b99bbbdbcbebab7448287b55aa111666f0d0102755082a5e"}
Oct 30 04:46:47.354271 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:47.353800    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn"
Oct 30 04:46:47.401759 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:47.401710    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn" podStartSLOduration=2.087178347 podStartE2EDuration="2.401695606s" totalImagesPullingTime="314.517259ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:46:47.401080691 +0000 UTC m=+2926.813476263" watchObservedRunningTime="2026-10-30 04:46:47.401695606 +0000 UTC m=+2926.814091178"
Oct 30 04:46:48.452161 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.452133    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"]
Oct 30 04:46:48.456021 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.455991    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.474374 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.474348    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-721c85-openshift-default-dockercfg-62hls\""
Oct 30 04:46:48.492832 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.492803    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"]
Oct 30 04:46:48.619190 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.619150    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-podinfo\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.619470 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.619446    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klmvz\" (UniqueName: \"kubernetes.io/projected/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-kube-api-access-klmvz\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.619644 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.619628    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-workload-socket\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.619810 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.619792    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-token\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.619957 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.619942    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-credential-socket\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.635264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.620078    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-data\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.635586 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.635556    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istiod-ca-cert\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.635785 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.635765    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-envoy\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.635965 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.635948    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-workload-certs\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.746269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.739426    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istiod-ca-cert\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.746269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.739510    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-envoy\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.746269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.739588    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-workload-certs\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.746269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.739613    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-podinfo\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.746269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.739642    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klmvz\" (UniqueName: \"kubernetes.io/projected/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-kube-api-access-klmvz\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.746269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.739718    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-workload-socket\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.746269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.739782    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-token\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.746269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.739848    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-credential-socket\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.746269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.739898    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-data\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.746269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.740423    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-data\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.746269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.741067    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istiod-ca-cert\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.746269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.742723    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-workload-certs\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.746269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.743713    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-workload-socket\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.746269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.744048    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-credential-socket\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.781984 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.780453    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klmvz\" (UniqueName: \"kubernetes.io/projected/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-kube-api-access-klmvz\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.781984 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.781910    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-token\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.789291 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.783760    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-envoy\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:48.789912 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:48.789886    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-podinfo\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-fz988\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:49.070093 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:49.070004    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:49.399414 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:49.399390    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"]
Oct 30 04:46:49.401625 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:46:49.401591    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c6bfea_f0bb_4ba6_b4fd_7e0979611108.slice/crio-c4ecad5285b15b26f20c6051ecfb9700d11e307e0661070e021dd40663a428e6 WatchSource:0}: Error finding container c4ecad5285b15b26f20c6051ecfb9700d11e307e0661070e021dd40663a428e6: Status 404 returned error can't find the container with id c4ecad5285b15b26f20c6051ecfb9700d11e307e0661070e021dd40663a428e6
Oct 30 04:46:49.404205 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:49.404167    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:46:49.404323 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:49.404252    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:46:49.404323 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:49.404284    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:46:50.374004 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:50.373971    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988" event={"ID":"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108","Type":"ContainerStarted","Data":"8b6afe42671ca634326bde5bbdaacbef0d1b81944a8432c545a10a5f313e68c5"}
Oct 30 04:46:50.374428 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:50.374024    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988" event={"ID":"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108","Type":"ContainerStarted","Data":"c4ecad5285b15b26f20c6051ecfb9700d11e307e0661070e021dd40663a428e6"}
Oct 30 04:46:50.431097 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:50.431046    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988" podStartSLOduration=2.431034356 podStartE2EDuration="2.431034356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:46:50.428044491 +0000 UTC m=+2929.840440065" watchObservedRunningTime="2026-10-30 04:46:50.431034356 +0000 UTC m=+2929.843429930"
Oct 30 04:46:51.070206 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:51.070175    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:51.075849 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:51.075823    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:51.380360 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:51.380286    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:51.381252 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:51.381223    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:46:53.366951 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:46:53.366918    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn"
Oct 30 04:46:55.608553 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:46:55.608508    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:47:17.639306 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:17.639266    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:47:25.615595 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:47:25.615559    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:47:27.168376 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:27.168338    4874 patch_prober.go:29] interesting pod/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 04:47:27.168865 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:27.168418    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l" podUID="c7abcd90-6f30-44af-bdc4-760820351783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 04:47:28.636551 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:28.636517    4874 generic.go:356] "Generic (PLEG): container finished" podID="c7abcd90-6f30-44af-bdc4-760820351783" containerID="050398fb1f680c4957e138b470dc3a80d22c3535671abfae8715fb6ce1a29536" exitCode=0
Oct 30 04:47:28.636993 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:28.636582    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l" event={"ID":"c7abcd90-6f30-44af-bdc4-760820351783","Type":"ContainerDied","Data":"050398fb1f680c4957e138b470dc3a80d22c3535671abfae8715fb6ce1a29536"}
Oct 30 04:47:28.637076 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:28.637057    4874 scope.go:122] "RemoveContainer" containerID="050398fb1f680c4957e138b470dc3a80d22c3535671abfae8715fb6ce1a29536"
Oct 30 04:47:28.638673 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:28.638640    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:47:28.638797 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:28.638729    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:47:28.638797 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:28.638781    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:47:29.644994 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:29.644958    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l" event={"ID":"c7abcd90-6f30-44af-bdc4-760820351783","Type":"ContainerStarted","Data":"341e7847bb54be5f574ccbd42de23924ca1fd62481d274d019f0b42d1a3f0bd0"}
Oct 30 04:47:30.423305 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:30.423276    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:47:30.423486 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:30.423441    4874 patch_prober.go:29] interesting pod/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 04:47:30.423537 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:30.423485    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l" podUID="c7abcd90-6f30-44af-bdc4-760820351783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 04:47:31.424180 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:31.424143    4874 patch_prober.go:29] interesting pod/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 04:47:31.424599 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:31.424196    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l" podUID="c7abcd90-6f30-44af-bdc4-760820351783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 04:47:32.424411 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:32.424373    4874 patch_prober.go:29] interesting pod/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 04:47:32.424782 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:32.424429    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l" podUID="c7abcd90-6f30-44af-bdc4-760820351783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 04:47:33.423559 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:33.423524    4874 patch_prober.go:29] interesting pod/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 04:47:33.423764 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:33.423583    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l" podUID="c7abcd90-6f30-44af-bdc4-760820351783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 04:47:34.433259 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:34.425773    4874 patch_prober.go:29] interesting pod/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 04:47:34.433259 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:34.425834    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l" podUID="c7abcd90-6f30-44af-bdc4-760820351783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 04:47:35.672627 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:35.667591    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:47:35.672627 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:35.668875    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:47:35.672627 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:35.672426    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 04:47:48.600262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:48.593596    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:47:55.613706 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:47:55.613668    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:47:58.878312 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:58.878280    4874 generic.go:356] "Generic (PLEG): container finished" podID="e7c6bfea-f0bb-4ba6-b4fd-7e0979611108" containerID="8b6afe42671ca634326bde5bbdaacbef0d1b81944a8432c545a10a5f313e68c5" exitCode=0
Oct 30 04:47:58.878648 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:58.878324    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988" event={"ID":"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108","Type":"ContainerDied","Data":"8b6afe42671ca634326bde5bbdaacbef0d1b81944a8432c545a10a5f313e68c5"}
Oct 30 04:47:58.878796 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:58.878778    4874 scope.go:122] "RemoveContainer" containerID="8b6afe42671ca634326bde5bbdaacbef0d1b81944a8432c545a10a5f313e68c5"
Oct 30 04:47:58.880687 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:58.880657    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:47:58.880761 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:58.880733    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:47:58.880799 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:58.880764    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:47:59.894906 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:47:59.894865    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988" event={"ID":"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108","Type":"ContainerStarted","Data":"cfc1c0fd932bc824d0bbcec1637ebc32b84272177d7b2c5263e7116e7771b449"}
Oct 30 04:48:00.082676 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:48:00.082641    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:48:00.083020 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:48:00.082997    4874 patch_prober.go:29] interesting pod/e2e-autores-721c85-openshift-default-7dfccc8995-fz988 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 30 04:48:00.083173 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:48:00.083149    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988" podUID="e7c6bfea-f0bb-4ba6-b4fd-7e0979611108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 30 04:48:01.031978 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:48:01.031949    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 04:48:01.085261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:48:01.082338    4874 patch_prober.go:29] interesting pod/e2e-autores-721c85-openshift-default-7dfccc8995-fz988 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 30 04:48:01.085261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:48:01.082409    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988" podUID="e7c6bfea-f0bb-4ba6-b4fd-7e0979611108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 30 04:48:01.091485 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:48:01.091460    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 04:48:02.083262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:48:02.082305    4874 patch_prober.go:29] interesting pod/e2e-autores-721c85-openshift-default-7dfccc8995-fz988 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 30 04:48:02.083262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:48:02.082366    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988" podUID="e7c6bfea-f0bb-4ba6-b4fd-7e0979611108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 30 04:48:03.080263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:48:03.077547    4874 patch_prober.go:29] interesting pod/e2e-autores-721c85-openshift-default-7dfccc8995-fz988 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 30 04:48:03.080263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:48:03.077621    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988" podUID="e7c6bfea-f0bb-4ba6-b4fd-7e0979611108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 30 04:48:04.078343 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:48:04.077872    4874 patch_prober.go:29] interesting pod/e2e-autores-721c85-openshift-default-7dfccc8995-fz988 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 30 04:48:04.078343 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:48:04.077942    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988" podUID="e7c6bfea-f0bb-4ba6-b4fd-7e0979611108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 30 04:48:05.078713 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:48:05.078671    4874 patch_prober.go:29] interesting pod/e2e-autores-721c85-openshift-default-7dfccc8995-fz988 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 30 04:48:05.079374 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:48:05.079341    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988" podUID="e7c6bfea-f0bb-4ba6-b4fd-7e0979611108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 30 04:48:06.076074 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:48:06.076042    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:48:06.076516 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:48:06.076492    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:48:06.077277 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:48:06.077258    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:48:16.776303 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:48:16.776218    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:48:25.609920 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:48:25.609883    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:48:55.610176 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:48:55.610133    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:48:58.798830 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:48:58.798796    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:49:18.883127 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:18.883089    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"]
Oct 30 04:49:18.886972 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:18.886955    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:18.901058 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:18.901033    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-91705f-openshift-default-dockercfg-6jv45\""
Oct 30 04:49:18.915497 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:18.915464    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"]
Oct 30 04:49:18.961605 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:18.961577    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-token\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:18.961757 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:18.961621    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-workload-socket\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:18.961757 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:18.961699    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3db5246-707f-4592-8ea0-f09ede62b5af-istiod-ca-cert\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:18.961892 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:18.961835    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-podinfo\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:18.961892 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:18.961866    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp8nf\" (UniqueName: \"kubernetes.io/projected/f3db5246-707f-4592-8ea0-f09ede62b5af-kube-api-access-sp8nf\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:18.961980 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:18.961944    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-workload-certs\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:18.962020 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:18.961991    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-envoy\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:18.962064 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:18.962020    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-credential-socket\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:18.962113 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:18.962093    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-data\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:19.063431 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:19.063403    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-podinfo\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:19.063592 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:19.063436    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp8nf\" (UniqueName: \"kubernetes.io/projected/f3db5246-707f-4592-8ea0-f09ede62b5af-kube-api-access-sp8nf\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:19.063592 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:19.063469    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-workload-certs\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:19.063592 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:19.063495    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-envoy\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:19.063592 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:19.063521    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-credential-socket\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:19.063592 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:19.063574    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-data\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:19.063858 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:19.063643    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-token\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:19.063858 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:19.063722    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-workload-socket\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:19.063858 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:19.063761    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3db5246-707f-4592-8ea0-f09ede62b5af-istiod-ca-cert\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:19.064036 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:19.063968    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-workload-certs\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:19.064036 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:19.063976    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-data\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:19.064145 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:19.064099    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-workload-socket\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:19.064281 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:19.064227    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-credential-socket\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:19.064478 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:19.064458    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3db5246-707f-4592-8ea0-f09ede62b5af-istiod-ca-cert\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:19.065876 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:19.065851    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-envoy\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:19.065971 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:19.065955    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-podinfo\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:19.086091 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:19.086065    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-token\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:19.086180 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:19.086135    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp8nf\" (UniqueName: \"kubernetes.io/projected/f3db5246-707f-4592-8ea0-f09ede62b5af-kube-api-access-sp8nf\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:19.200846 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:19.200781    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:19.555773 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:19.555744    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"]
Oct 30 04:49:19.558159 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:49:19.558132    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3db5246_707f_4592_8ea0_f09ede62b5af.slice/crio-c76db20065bfacbd6af05ada1fa0793e3db1253bed3eec6594ecebe3fe607e54 WatchSource:0}: Error finding container c76db20065bfacbd6af05ada1fa0793e3db1253bed3eec6594ecebe3fe607e54: Status 404 returned error can't find the container with id c76db20065bfacbd6af05ada1fa0793e3db1253bed3eec6594ecebe3fe607e54
Oct 30 04:49:19.560794 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:19.560760    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:49:19.560889 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:19.560825    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:49:19.560889 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:19.560857    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:49:20.499583 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:20.499543    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr" event={"ID":"f3db5246-707f-4592-8ea0-f09ede62b5af","Type":"ContainerStarted","Data":"9c4981967ec74204636dad388e0b7127cbd40c2f21ec0f1ca64f8a3d2f6010f7"}
Oct 30 04:49:20.500019 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:20.499597    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr" event={"ID":"f3db5246-707f-4592-8ea0-f09ede62b5af","Type":"ContainerStarted","Data":"c76db20065bfacbd6af05ada1fa0793e3db1253bed3eec6594ecebe3fe607e54"}
Oct 30 04:49:20.556446 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:20.556396    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr" podStartSLOduration=2.556382966 podStartE2EDuration="2.556382966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:49:20.554098869 +0000 UTC m=+3079.966494442" watchObservedRunningTime="2026-10-30 04:49:20.556382966 +0000 UTC m=+3079.968778539"
Oct 30 04:49:21.210263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:21.207105    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:21.210263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:21.209108    4874 patch_prober.go:29] interesting pod/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 30 04:49:21.210263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:21.209174    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr" podUID="f3db5246-707f-4592-8ea0-f09ede62b5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 30 04:49:22.201582 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:22.201550    4874 patch_prober.go:29] interesting pod/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 30 04:49:22.202492 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:22.202446    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr" podUID="f3db5246-707f-4592-8ea0-f09ede62b5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 30 04:49:23.202092 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:23.202059    4874 patch_prober.go:29] interesting pod/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 30 04:49:23.202546 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:23.202111    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr" podUID="f3db5246-707f-4592-8ea0-f09ede62b5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 30 04:49:24.202064 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:24.202027    4874 patch_prober.go:29] interesting pod/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 30 04:49:24.202285 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:24.202085    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr" podUID="f3db5246-707f-4592-8ea0-f09ede62b5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 30 04:49:25.208933 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:25.208896    4874 patch_prober.go:29] interesting pod/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 30 04:49:25.209666 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:25.209633    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr" podUID="f3db5246-707f-4592-8ea0-f09ede62b5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 30 04:49:25.613252 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:49:25.613205    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:49:26.338746 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:26.338707    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:26.339537 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:26.339502    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:26.340453 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:26.340435    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 04:49:39.491093 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:39.491046    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:49:40.892785 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:40.892745    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"]
Oct 30 04:49:40.893622 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:40.893563    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988" podUID="e7c6bfea-f0bb-4ba6-b4fd-7e0979611108" containerName="istio-proxy" containerID="cri-o://cfc1c0fd932bc824d0bbcec1637ebc32b84272177d7b2c5263e7116e7771b449" gracePeriod=30
Oct 30 04:49:46.180413 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.180350    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:49:46.221927 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.221896    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-data\") pod \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") "
Oct 30 04:49:46.222078 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.222004    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-token\" (UniqueName: \"kubernetes.io/projected/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-token\") pod \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") "
Oct 30 04:49:46.222078 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.222033    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-credential-socket\") pod \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") "
Oct 30 04:49:46.222078 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.222072    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-workload-socket\") pod \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") "
Oct 30 04:49:46.222286 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.222131    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-data" pod "e7c6bfea-f0bb-4ba6-b4fd-7e0979611108" (UID: "e7c6bfea-f0bb-4ba6-b4fd-7e0979611108"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:49:46.222286 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.222186    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-envoy\") pod \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") "
Oct 30 04:49:46.222434 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.222295    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-workload-certs\") pod \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") "
Oct 30 04:49:46.222434 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.222361    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-credential-socket" pod "e7c6bfea-f0bb-4ba6-b4fd-7e0979611108" (UID: "e7c6bfea-f0bb-4ba6-b4fd-7e0979611108"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:49:46.222434 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.222417    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-workload-socket" pod "e7c6bfea-f0bb-4ba6-b4fd-7e0979611108" (UID: "e7c6bfea-f0bb-4ba6-b4fd-7e0979611108"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:49:46.222584 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.222440    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-kube-api-access-klmvz\" (UniqueName: \"kubernetes.io/projected/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-kube-api-access-klmvz\") pod \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") "
Oct 30 04:49:46.222584 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.222500    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-podinfo\") pod \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") "
Oct 30 04:49:46.222584 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.222505    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-workload-certs" pod "e7c6bfea-f0bb-4ba6-b4fd-7e0979611108" (UID: "e7c6bfea-f0bb-4ba6-b4fd-7e0979611108"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:49:46.222584 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.222552    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istiod-ca-cert\") pod \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\" (UID: \"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108\") "
Oct 30 04:49:46.222951 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.222927    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istiod-ca-cert" pod "e7c6bfea-f0bb-4ba6-b4fd-7e0979611108" (UID: "e7c6bfea-f0bb-4ba6-b4fd-7e0979611108"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:49:46.223080 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.223052    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:49:46.223080 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.223076    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:49:46.223301 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.223090    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:49:46.223301 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.223100    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:49:46.223301 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.223111    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:49:46.224347 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.224327    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-envoy" pod "e7c6bfea-f0bb-4ba6-b4fd-7e0979611108" (UID: "e7c6bfea-f0bb-4ba6-b4fd-7e0979611108"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:49:46.224600 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.224586    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-kube-api-access-klmvz" pod "e7c6bfea-f0bb-4ba6-b4fd-7e0979611108" (UID: "e7c6bfea-f0bb-4ba6-b4fd-7e0979611108"). InnerVolumeSpecName "kube-api-access-klmvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:49:46.224708 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.224685    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-token" pod "e7c6bfea-f0bb-4ba6-b4fd-7e0979611108" (UID: "e7c6bfea-f0bb-4ba6-b4fd-7e0979611108"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:49:46.224779 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.224766    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-podinfo" pod "e7c6bfea-f0bb-4ba6-b4fd-7e0979611108" (UID: "e7c6bfea-f0bb-4ba6-b4fd-7e0979611108"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:49:46.324458 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.324429    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:49:46.324458 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.324454    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:49:46.324458 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.324465    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klmvz\" (UniqueName: \"kubernetes.io/projected/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-kube-api-access-klmvz\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:49:46.324678 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.324475    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:49:46.746726 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.746686    4874 generic.go:356] "Generic (PLEG): container finished" podID="e7c6bfea-f0bb-4ba6-b4fd-7e0979611108" containerID="cfc1c0fd932bc824d0bbcec1637ebc32b84272177d7b2c5263e7116e7771b449" exitCode=0
Oct 30 04:49:46.746726 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.746724    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"
Oct 30 04:49:46.746948 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.746761    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988" event={"ID":"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108","Type":"ContainerDied","Data":"cfc1c0fd932bc824d0bbcec1637ebc32b84272177d7b2c5263e7116e7771b449"}
Oct 30 04:49:46.746948 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.746819    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988" event={"ID":"e7c6bfea-f0bb-4ba6-b4fd-7e0979611108","Type":"ContainerDied","Data":"c4ecad5285b15b26f20c6051ecfb9700d11e307e0661070e021dd40663a428e6"}
Oct 30 04:49:46.746948 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.746848    4874 scope.go:122] "RemoveContainer" containerID="cfc1c0fd932bc824d0bbcec1637ebc32b84272177d7b2c5263e7116e7771b449"
Oct 30 04:49:46.758400 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.758382    4874 scope.go:122] "RemoveContainer" containerID="8b6afe42671ca634326bde5bbdaacbef0d1b81944a8432c545a10a5f313e68c5"
Oct 30 04:49:46.768265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.768225    4874 scope.go:122] "RemoveContainer" containerID="cfc1c0fd932bc824d0bbcec1637ebc32b84272177d7b2c5263e7116e7771b449"
Oct 30 04:49:46.768595 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:49:46.768577    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfc1c0fd932bc824d0bbcec1637ebc32b84272177d7b2c5263e7116e7771b449\": container with ID starting with cfc1c0fd932bc824d0bbcec1637ebc32b84272177d7b2c5263e7116e7771b449 not found: ID does not exist" containerID="cfc1c0fd932bc824d0bbcec1637ebc32b84272177d7b2c5263e7116e7771b449"
Oct 30 04:49:46.768699 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.768605    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfc1c0fd932bc824d0bbcec1637ebc32b84272177d7b2c5263e7116e7771b449"} err="failed to get container status \"cfc1c0fd932bc824d0bbcec1637ebc32b84272177d7b2c5263e7116e7771b449\": rpc error: code = NotFound desc = could not find container \"cfc1c0fd932bc824d0bbcec1637ebc32b84272177d7b2c5263e7116e7771b449\": container with ID starting with cfc1c0fd932bc824d0bbcec1637ebc32b84272177d7b2c5263e7116e7771b449 not found: ID does not exist"
Oct 30 04:49:46.768699 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.768625    4874 scope.go:122] "RemoveContainer" containerID="8b6afe42671ca634326bde5bbdaacbef0d1b81944a8432c545a10a5f313e68c5"
Oct 30 04:49:46.768865 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:49:46.768847    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b6afe42671ca634326bde5bbdaacbef0d1b81944a8432c545a10a5f313e68c5\": container with ID starting with 8b6afe42671ca634326bde5bbdaacbef0d1b81944a8432c545a10a5f313e68c5 not found: ID does not exist" containerID="8b6afe42671ca634326bde5bbdaacbef0d1b81944a8432c545a10a5f313e68c5"
Oct 30 04:49:46.768922 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.768876    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b6afe42671ca634326bde5bbdaacbef0d1b81944a8432c545a10a5f313e68c5"} err="failed to get container status \"8b6afe42671ca634326bde5bbdaacbef0d1b81944a8432c545a10a5f313e68c5\": rpc error: code = NotFound desc = could not find container \"8b6afe42671ca634326bde5bbdaacbef0d1b81944a8432c545a10a5f313e68c5\": container with ID starting with 8b6afe42671ca634326bde5bbdaacbef0d1b81944a8432c545a10a5f313e68c5 not found: ID does not exist"
Oct 30 04:49:46.797136 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.797115    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"]
Oct 30 04:49:46.808132 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:46.808109    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-fz988"]
Oct 30 04:49:48.731327 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:48.731295    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7c6bfea-f0bb-4ba6-b4fd-7e0979611108" path="/var/lib/kubelet/pods/e7c6bfea-f0bb-4ba6-b4fd-7e0979611108/volumes"
Oct 30 04:49:49.225029 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:49.225004    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff"]
Oct 30 04:49:49.225710 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:49.225686    4874 cpu_manager.go:414] "containerMap: removing container" podUID="e7c6bfea-f0bb-4ba6-b4fd-7e0979611108" containerName="istio-proxy"
Oct 30 04:49:49.225710 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:49.225709    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7c6bfea-f0bb-4ba6-b4fd-7e0979611108" containerName="istio-proxy"
Oct 30 04:49:49.225881 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:49.225734    4874 cpu_manager.go:414] "containerMap: removing container" podUID="e7c6bfea-f0bb-4ba6-b4fd-7e0979611108" containerName="istio-proxy"
Oct 30 04:49:49.225881 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:49.225741    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7c6bfea-f0bb-4ba6-b4fd-7e0979611108" containerName="istio-proxy"
Oct 30 04:49:49.225988 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:49.225911    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7c6bfea-f0bb-4ba6-b4fd-7e0979611108" containerName="istio-proxy"
Oct 30 04:49:49.225988 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:49.225926    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7c6bfea-f0bb-4ba6-b4fd-7e0979611108" containerName="istio-proxy"
Oct 30 04:49:49.230860 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:49.230839    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff"
Oct 30 04:49:49.256142 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:49.256119    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff"]
Oct 30 04:49:49.257826 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:49.257799    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ac6eaed-819d-4ce2-b299-b7228312da81-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff\" (UID: \"2ac6eaed-819d-4ce2-b299-b7228312da81\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff"
Oct 30 04:49:49.257953 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:49.257933    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57bf4\" (UniqueName: \"kubernetes.io/projected/2ac6eaed-819d-4ce2-b299-b7228312da81-kube-api-access-57bf4\") pod \"payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff\" (UID: \"2ac6eaed-819d-4ce2-b299-b7228312da81\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff"
Oct 30 04:49:49.264512 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:49.264492    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c0d6e7\""
Oct 30 04:49:49.264592 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:49.264512    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-c0d6e7-dockercfg-xxc8s\""
Oct 30 04:49:49.379263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:49.365513    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57bf4\" (UniqueName: \"kubernetes.io/projected/2ac6eaed-819d-4ce2-b299-b7228312da81-kube-api-access-57bf4\") pod \"payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff\" (UID: \"2ac6eaed-819d-4ce2-b299-b7228312da81\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff"
Oct 30 04:49:49.379788 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:49.379756    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ac6eaed-819d-4ce2-b299-b7228312da81-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff\" (UID: \"2ac6eaed-819d-4ce2-b299-b7228312da81\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff"
Oct 30 04:49:49.391269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:49.388056    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ac6eaed-819d-4ce2-b299-b7228312da81-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff\" (UID: \"2ac6eaed-819d-4ce2-b299-b7228312da81\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff"
Oct 30 04:49:49.432313 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:49.432277    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57bf4\" (UniqueName: \"kubernetes.io/projected/2ac6eaed-819d-4ce2-b299-b7228312da81-kube-api-access-57bf4\") pod \"payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff\" (UID: \"2ac6eaed-819d-4ce2-b299-b7228312da81\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff"
Oct 30 04:49:49.544423 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:49.544341    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff"
Oct 30 04:49:49.847039 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:49.847006    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff"]
Oct 30 04:49:49.848885 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:49:49.848856    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac6eaed_819d_4ce2_b299_b7228312da81.slice/crio-942f0bf3cf95d381bded240bb5c5ec7b59b2333444886ec742132983b8a4db4e WatchSource:0}: Error finding container 942f0bf3cf95d381bded240bb5c5ec7b59b2333444886ec742132983b8a4db4e: Status 404 returned error can't find the container with id 942f0bf3cf95d381bded240bb5c5ec7b59b2333444886ec742132983b8a4db4e
Oct 30 04:49:50.774738 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:50.774701    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff" event={"ID":"2ac6eaed-819d-4ce2-b299-b7228312da81","Type":"ContainerStarted","Data":"e00621fb17f97b000e874824cb41d4b196741af3e00781a6213922ffe02a8532"}
Oct 30 04:49:50.774906 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:50.774767    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff" event={"ID":"2ac6eaed-819d-4ce2-b299-b7228312da81","Type":"ContainerStarted","Data":"942f0bf3cf95d381bded240bb5c5ec7b59b2333444886ec742132983b8a4db4e"}
Oct 30 04:49:50.774906 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:50.774797    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff"
Oct 30 04:49:50.817920 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:49:50.817867    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff" podStartSLOduration=1.817854026 podStartE2EDuration="1.817854026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:49:50.815279246 +0000 UTC m=+3110.227674818" watchObservedRunningTime="2026-10-30 04:49:50.817854026 +0000 UTC m=+3110.230249598"
Oct 30 04:49:55.610369 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:49:55.610326    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:50:01.783753 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:01.783725    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff"
Oct 30 04:50:23.587704 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:23.587673    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc"]
Oct 30 04:50:23.591927 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:23.591909    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc"
Oct 30 04:50:23.610415 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:23.610392    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-503804\"/\"payload-processing-external-model-plugins-e2e-extmod-b-503804\""
Oct 30 04:50:23.621740 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:23.621716    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-503804\"/\"agc-cred-1b38359d0333\""
Oct 30 04:50:23.621885 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:23.621833    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-503804\"/\"kube-root-ca.crt\""
Oct 30 04:50:23.621957 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:23.621922    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-503804\"/\"openshift-service-ca.crt\""
Oct 30 04:50:23.622056 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:23.622029    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-503804\"/\"agc-cred-259031349671\""
Oct 30 04:50:23.622148 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:23.622092    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-503804\"/\"routing-overlay\""
Oct 30 04:50:23.622299 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:23.622276    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-503804\"/\"payload-processing-external-model-e2e--0b0ba164-dockercfg-xpm8t\""
Oct 30 04:50:23.626827 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:23.626805    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc"]
Oct 30 04:50:23.670711 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:23.670683    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17e85c62-1576-48cc-a5e0-3353a5e0e904-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc\" (UID: \"17e85c62-1576-48cc-a5e0-3353a5e0e904\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc"
Oct 30 04:50:23.670811 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:23.670720    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6jkc\" (UniqueName: \"kubernetes.io/projected/17e85c62-1576-48cc-a5e0-3353a5e0e904-kube-api-access-n6jkc\") pod \"payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc\" (UID: \"17e85c62-1576-48cc-a5e0-3353a5e0e904\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc"
Oct 30 04:50:23.670884 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:23.670866    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e85c62-1576-48cc-a5e0-3353a5e0e904-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc\" (UID: \"17e85c62-1576-48cc-a5e0-3353a5e0e904\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc"
Oct 30 04:50:23.670933 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:23.670918    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17e85c62-1576-48cc-a5e0-3353a5e0e904-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc\" (UID: \"17e85c62-1576-48cc-a5e0-3353a5e0e904\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc"
Oct 30 04:50:23.772510 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:23.772480    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17e85c62-1576-48cc-a5e0-3353a5e0e904-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc\" (UID: \"17e85c62-1576-48cc-a5e0-3353a5e0e904\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc"
Oct 30 04:50:23.772652 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:23.772514    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6jkc\" (UniqueName: \"kubernetes.io/projected/17e85c62-1576-48cc-a5e0-3353a5e0e904-kube-api-access-n6jkc\") pod \"payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc\" (UID: \"17e85c62-1576-48cc-a5e0-3353a5e0e904\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc"
Oct 30 04:50:23.772652 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:23.772574    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e85c62-1576-48cc-a5e0-3353a5e0e904-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc\" (UID: \"17e85c62-1576-48cc-a5e0-3353a5e0e904\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc"
Oct 30 04:50:23.772652 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:23.772603    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17e85c62-1576-48cc-a5e0-3353a5e0e904-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc\" (UID: \"17e85c62-1576-48cc-a5e0-3353a5e0e904\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc"
Oct 30 04:50:23.773110 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:23.773083    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17e85c62-1576-48cc-a5e0-3353a5e0e904-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc\" (UID: \"17e85c62-1576-48cc-a5e0-3353a5e0e904\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc"
Oct 30 04:50:23.773286 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:23.773268    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e85c62-1576-48cc-a5e0-3353a5e0e904-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc\" (UID: \"17e85c62-1576-48cc-a5e0-3353a5e0e904\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc"
Oct 30 04:50:23.775100 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:23.775075    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17e85c62-1576-48cc-a5e0-3353a5e0e904-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc\" (UID: \"17e85c62-1576-48cc-a5e0-3353a5e0e904\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc"
Oct 30 04:50:23.794784 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:23.794760    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6jkc\" (UniqueName: \"kubernetes.io/projected/17e85c62-1576-48cc-a5e0-3353a5e0e904-kube-api-access-n6jkc\") pod \"payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc\" (UID: \"17e85c62-1576-48cc-a5e0-3353a5e0e904\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc"
Oct 30 04:50:23.952704 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:23.952627    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc"
Oct 30 04:50:24.272037 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:24.272010    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc"]
Oct 30 04:50:24.276328 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:50:24.276299    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e85c62_1576_48cc_a5e0_3353a5e0e904.slice/crio-6bed2336ce8f0f16bc24a1b4b67f79f63231299cba1d4f9ed9342c2b20b5de76 WatchSource:0}: Error finding container 6bed2336ce8f0f16bc24a1b4b67f79f63231299cba1d4f9ed9342c2b20b5de76: Status 404 returned error can't find the container with id 6bed2336ce8f0f16bc24a1b4b67f79f63231299cba1d4f9ed9342c2b20b5de76
Oct 30 04:50:24.497396 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:24.497358    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5"]
Oct 30 04:50:24.501381 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:24.501357    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5"
Oct 30 04:50:24.515110 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:24.515092    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-c0d6e7-serving-cert\""
Oct 30 04:50:24.522489 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:24.522372    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5"]
Oct 30 04:50:24.586407 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:24.586381    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a849651f-2575-40e3-9f53-3c2b057075ce-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5\" (UID: \"a849651f-2575-40e3-9f53-3c2b057075ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5"
Oct 30 04:50:24.586570 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:24.586446    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a849651f-2575-40e3-9f53-3c2b057075ce-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5\" (UID: \"a849651f-2575-40e3-9f53-3c2b057075ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5"
Oct 30 04:50:24.586570 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:24.586499    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a849651f-2575-40e3-9f53-3c2b057075ce-metrics-certs\") pod \"maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5\" (UID: \"a849651f-2575-40e3-9f53-3c2b057075ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5"
Oct 30 04:50:24.586661 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:24.586582    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxzfp\" (UniqueName: \"kubernetes.io/projected/a849651f-2575-40e3-9f53-3c2b057075ce-kube-api-access-qxzfp\") pod \"maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5\" (UID: \"a849651f-2575-40e3-9f53-3c2b057075ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5"
Oct 30 04:50:24.687530 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:24.687497    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a849651f-2575-40e3-9f53-3c2b057075ce-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5\" (UID: \"a849651f-2575-40e3-9f53-3c2b057075ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5"
Oct 30 04:50:24.687943 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:24.687574    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a849651f-2575-40e3-9f53-3c2b057075ce-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5\" (UID: \"a849651f-2575-40e3-9f53-3c2b057075ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5"
Oct 30 04:50:24.687943 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:24.687599    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a849651f-2575-40e3-9f53-3c2b057075ce-metrics-certs\") pod \"maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5\" (UID: \"a849651f-2575-40e3-9f53-3c2b057075ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5"
Oct 30 04:50:24.687943 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:24.687632    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxzfp\" (UniqueName: \"kubernetes.io/projected/a849651f-2575-40e3-9f53-3c2b057075ce-kube-api-access-qxzfp\") pod \"maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5\" (UID: \"a849651f-2575-40e3-9f53-3c2b057075ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5"
Oct 30 04:50:24.688325 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:24.688301    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a849651f-2575-40e3-9f53-3c2b057075ce-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5\" (UID: \"a849651f-2575-40e3-9f53-3c2b057075ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5"
Oct 30 04:50:24.690068 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:24.690045    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a849651f-2575-40e3-9f53-3c2b057075ce-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5\" (UID: \"a849651f-2575-40e3-9f53-3c2b057075ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5"
Oct 30 04:50:24.690158 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:24.690107    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a849651f-2575-40e3-9f53-3c2b057075ce-metrics-certs\") pod \"maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5\" (UID: \"a849651f-2575-40e3-9f53-3c2b057075ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5"
Oct 30 04:50:24.709681 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:24.709661    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxzfp\" (UniqueName: \"kubernetes.io/projected/a849651f-2575-40e3-9f53-3c2b057075ce-kube-api-access-qxzfp\") pod \"maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5\" (UID: \"a849651f-2575-40e3-9f53-3c2b057075ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5"
Oct 30 04:50:24.815434 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:24.813979    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5"
Oct 30 04:50:25.014203 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:25.014169    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc" event={"ID":"17e85c62-1576-48cc-a5e0-3353a5e0e904","Type":"ContainerStarted","Data":"029e217d6c8d658a6e78bb9b7fc144033749d88199b60d94fc10179e177a5e6f"}
Oct 30 04:50:25.014436 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:25.014227    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc" event={"ID":"17e85c62-1576-48cc-a5e0-3353a5e0e904","Type":"ContainerStarted","Data":"6bed2336ce8f0f16bc24a1b4b67f79f63231299cba1d4f9ed9342c2b20b5de76"}
Oct 30 04:50:25.014436 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:25.014276    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc"
Oct 30 04:50:25.053363 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:25.053314    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc" podStartSLOduration=2.053297933 podStartE2EDuration="2.053297933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:50:25.053105572 +0000 UTC m=+3144.465501165" watchObservedRunningTime="2026-10-30 04:50:25.053297933 +0000 UTC m=+3144.465693507"
Oct 30 04:50:25.120869 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:25.120843    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5"]
Oct 30 04:50:25.122608 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:50:25.122580    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda849651f_2575_40e3_9f53_3c2b057075ce.slice/crio-13f94d77ff4c302c19867b3b3e87c3ba0ecfb44ea44ab89b4ee1e380f95136a0 WatchSource:0}: Error finding container 13f94d77ff4c302c19867b3b3e87c3ba0ecfb44ea44ab89b4ee1e380f95136a0: Status 404 returned error can't find the container with id 13f94d77ff4c302c19867b3b3e87c3ba0ecfb44ea44ab89b4ee1e380f95136a0
Oct 30 04:50:25.123975 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:25.123955    4874 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 04:50:25.689146 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:50:25.689117    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:50:26.024199 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:26.024169    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5" event={"ID":"a849651f-2575-40e3-9f53-3c2b057075ce","Type":"ContainerStarted","Data":"62d8b40844774686e5bc85c02bbc6be065e7c2a07d34cace68443a968d78f653"}
Oct 30 04:50:26.024558 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:26.024218    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5" event={"ID":"a849651f-2575-40e3-9f53-3c2b057075ce","Type":"ContainerStarted","Data":"13f94d77ff4c302c19867b3b3e87c3ba0ecfb44ea44ab89b4ee1e380f95136a0"}
Oct 30 04:50:26.024558 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:26.024467    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5"
Oct 30 04:50:26.066328 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:26.066230    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5" podStartSLOduration=1.503699504 podStartE2EDuration="2.066217104s" totalImagesPullingTime="562.5176ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:50:26.063930447 +0000 UTC m=+3145.476326020" watchObservedRunningTime="2026-10-30 04:50:26.066217104 +0000 UTC m=+3145.478612677"
Oct 30 04:50:32.035003 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:32.034975    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5"
Oct 30 04:50:34.998156 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:34.998119    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:50:35.631432 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:35.631399    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc"]
Oct 30 04:50:35.634616 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:35.634591    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc"
Oct 30 04:50:40.632106 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:40.632048    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc" podUID="17e85c62-1576-48cc-a5e0-3353a5e0e904" containerName="payload-processing" containerID="cri-o://029e217d6c8d658a6e78bb9b7fc144033749d88199b60d94fc10179e177a5e6f" gracePeriod=25
Oct 30 04:50:40.897901 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:40.897836    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc"
Oct 30 04:50:40.951630 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:40.951601    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17e85c62-1576-48cc-a5e0-3353a5e0e904-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17e85c62-1576-48cc-a5e0-3353a5e0e904-routing-overlay-volume\") pod \"17e85c62-1576-48cc-a5e0-3353a5e0e904\" (UID: \"17e85c62-1576-48cc-a5e0-3353a5e0e904\") "
Oct 30 04:50:40.951818 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:40.951734    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17e85c62-1576-48cc-a5e0-3353a5e0e904-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e85c62-1576-48cc-a5e0-3353a5e0e904-plugins-config-volume\") pod \"17e85c62-1576-48cc-a5e0-3353a5e0e904\" (UID: \"17e85c62-1576-48cc-a5e0-3353a5e0e904\") "
Oct 30 04:50:40.951883 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:40.951846    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e85c62-1576-48cc-a5e0-3353a5e0e904-provider-credentials\" (UniqueName: \"kubernetes.io/projected/17e85c62-1576-48cc-a5e0-3353a5e0e904-provider-credentials\") pod \"17e85c62-1576-48cc-a5e0-3353a5e0e904\" (UID: \"17e85c62-1576-48cc-a5e0-3353a5e0e904\") "
Oct 30 04:50:40.951958 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:40.951935    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e85c62-1576-48cc-a5e0-3353a5e0e904-kube-api-access-n6jkc\" (UniqueName: \"kubernetes.io/projected/17e85c62-1576-48cc-a5e0-3353a5e0e904-kube-api-access-n6jkc\") pod \"17e85c62-1576-48cc-a5e0-3353a5e0e904\" (UID: \"17e85c62-1576-48cc-a5e0-3353a5e0e904\") "
Oct 30 04:50:40.952095 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:40.952036    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e85c62-1576-48cc-a5e0-3353a5e0e904-routing-overlay-volume" pod "17e85c62-1576-48cc-a5e0-3353a5e0e904" (UID: "17e85c62-1576-48cc-a5e0-3353a5e0e904"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:50:40.952302 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:40.952273    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e85c62-1576-48cc-a5e0-3353a5e0e904-plugins-config-volume" pod "17e85c62-1576-48cc-a5e0-3353a5e0e904" (UID: "17e85c62-1576-48cc-a5e0-3353a5e0e904"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:50:40.952549 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:40.952529    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e85c62-1576-48cc-a5e0-3353a5e0e904-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:50:40.952638 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:40.952553    4874 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17e85c62-1576-48cc-a5e0-3353a5e0e904-routing-overlay-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:50:40.954780 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:40.954754    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e85c62-1576-48cc-a5e0-3353a5e0e904-kube-api-access-n6jkc" pod "17e85c62-1576-48cc-a5e0-3353a5e0e904" (UID: "17e85c62-1576-48cc-a5e0-3353a5e0e904"). InnerVolumeSpecName "kube-api-access-n6jkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:50:40.954875 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:40.954792    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e85c62-1576-48cc-a5e0-3353a5e0e904-provider-credentials" pod "17e85c62-1576-48cc-a5e0-3353a5e0e904" (UID: "17e85c62-1576-48cc-a5e0-3353a5e0e904"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:50:41.056676 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:41.056645    4874 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17e85c62-1576-48cc-a5e0-3353a5e0e904-provider-credentials\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:50:41.056676 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:41.056672    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6jkc\" (UniqueName: \"kubernetes.io/projected/17e85c62-1576-48cc-a5e0-3353a5e0e904-kube-api-access-n6jkc\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:50:41.132048 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:41.132018    4874 generic.go:356] "Generic (PLEG): container finished" podID="17e85c62-1576-48cc-a5e0-3353a5e0e904" containerID="029e217d6c8d658a6e78bb9b7fc144033749d88199b60d94fc10179e177a5e6f" exitCode=0
Oct 30 04:50:41.132228 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:41.132065    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc"
Oct 30 04:50:41.132228 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:41.132107    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc" event={"ID":"17e85c62-1576-48cc-a5e0-3353a5e0e904","Type":"ContainerDied","Data":"029e217d6c8d658a6e78bb9b7fc144033749d88199b60d94fc10179e177a5e6f"}
Oct 30 04:50:41.132228 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:41.132166    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc" event={"ID":"17e85c62-1576-48cc-a5e0-3353a5e0e904","Type":"ContainerDied","Data":"6bed2336ce8f0f16bc24a1b4b67f79f63231299cba1d4f9ed9342c2b20b5de76"}
Oct 30 04:50:41.132228 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:41.132194    4874 scope.go:122] "RemoveContainer" containerID="029e217d6c8d658a6e78bb9b7fc144033749d88199b60d94fc10179e177a5e6f"
Oct 30 04:50:41.142521 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:41.142505    4874 scope.go:122] "RemoveContainer" containerID="029e217d6c8d658a6e78bb9b7fc144033749d88199b60d94fc10179e177a5e6f"
Oct 30 04:50:41.142794 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:50:41.142771    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"029e217d6c8d658a6e78bb9b7fc144033749d88199b60d94fc10179e177a5e6f\": container with ID starting with 029e217d6c8d658a6e78bb9b7fc144033749d88199b60d94fc10179e177a5e6f not found: ID does not exist" containerID="029e217d6c8d658a6e78bb9b7fc144033749d88199b60d94fc10179e177a5e6f"
Oct 30 04:50:41.142860 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:41.142801    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"029e217d6c8d658a6e78bb9b7fc144033749d88199b60d94fc10179e177a5e6f"} err="failed to get container status \"029e217d6c8d658a6e78bb9b7fc144033749d88199b60d94fc10179e177a5e6f\": rpc error: code = NotFound desc = could not find container \"029e217d6c8d658a6e78bb9b7fc144033749d88199b60d94fc10179e177a5e6f\": container with ID starting with 029e217d6c8d658a6e78bb9b7fc144033749d88199b60d94fc10179e177a5e6f not found: ID does not exist"
Oct 30 04:50:41.182360 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:41.182308    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc"]
Oct 30 04:50:41.188573 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:41.188552    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-8b9dgftxc"]
Oct 30 04:50:42.729999 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:50:42.729967    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17e85c62-1576-48cc-a5e0-3353a5e0e904" path="/var/lib/kubelet/pods/17e85c62-1576-48cc-a5e0-3353a5e0e904/volumes"
Oct 30 04:50:55.609997 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:50:55.609958    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:51:01.598593 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:01.598562    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:51:05.078287 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.073275    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd"]
Oct 30 04:51:05.078287 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.074086    4874 cpu_manager.go:414] "containerMap: removing container" podUID="17e85c62-1576-48cc-a5e0-3353a5e0e904" containerName="payload-processing"
Oct 30 04:51:05.078287 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.074103    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e85c62-1576-48cc-a5e0-3353a5e0e904" containerName="payload-processing"
Oct 30 04:51:05.078287 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.074318    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e85c62-1576-48cc-a5e0-3353a5e0e904" containerName="payload-processing"
Oct 30 04:51:05.090135 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.090089    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd"]
Oct 30 04:51:05.090533 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.090513    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd"
Oct 30 04:51:05.120479 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.120447    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6378cc\"/\"openshift-service-ca.crt\""
Oct 30 04:51:05.121360 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.121335    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6378cc\"/\"payload-processing-external-model-plugins-e2e-extmod-a-6378cc\""
Oct 30 04:51:05.121477 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.121419    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6378cc\"/\"routing-overlay\""
Oct 30 04:51:05.121535 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.121515    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6378cc\"/\"kube-root-ca.crt\""
Oct 30 04:51:05.169352 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.169298    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6378cc\"/\"e2e-cfl-8523f9-provider-cred\""
Oct 30 04:51:05.180623 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.178701    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6378cc\"/\"payload-processing-external-model-e2e--18641857-dockercfg-9hk4x\""
Oct 30 04:51:05.192091 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.191898    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trrrq\" (UniqueName: \"kubernetes.io/projected/eb760784-9728-48d0-9381-78fe27736f5f-kube-api-access-trrrq\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd\" (UID: \"eb760784-9728-48d0-9381-78fe27736f5f\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd"
Oct 30 04:51:05.192543 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.192514    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eb760784-9728-48d0-9381-78fe27736f5f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd\" (UID: \"eb760784-9728-48d0-9381-78fe27736f5f\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd"
Oct 30 04:51:05.192703 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.192688    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb760784-9728-48d0-9381-78fe27736f5f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd\" (UID: \"eb760784-9728-48d0-9381-78fe27736f5f\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd"
Oct 30 04:51:05.192900 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.192878    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eb760784-9728-48d0-9381-78fe27736f5f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd\" (UID: \"eb760784-9728-48d0-9381-78fe27736f5f\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd"
Oct 30 04:51:05.294057 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.294022    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eb760784-9728-48d0-9381-78fe27736f5f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd\" (UID: \"eb760784-9728-48d0-9381-78fe27736f5f\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd"
Oct 30 04:51:05.294221 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.294129    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trrrq\" (UniqueName: \"kubernetes.io/projected/eb760784-9728-48d0-9381-78fe27736f5f-kube-api-access-trrrq\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd\" (UID: \"eb760784-9728-48d0-9381-78fe27736f5f\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd"
Oct 30 04:51:05.294311 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.294293    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eb760784-9728-48d0-9381-78fe27736f5f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd\" (UID: \"eb760784-9728-48d0-9381-78fe27736f5f\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd"
Oct 30 04:51:05.294351 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.294335    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb760784-9728-48d0-9381-78fe27736f5f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd\" (UID: \"eb760784-9728-48d0-9381-78fe27736f5f\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd"
Oct 30 04:51:05.294794 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.294763    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eb760784-9728-48d0-9381-78fe27736f5f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd\" (UID: \"eb760784-9728-48d0-9381-78fe27736f5f\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd"
Oct 30 04:51:05.295147 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.295126    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb760784-9728-48d0-9381-78fe27736f5f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd\" (UID: \"eb760784-9728-48d0-9381-78fe27736f5f\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd"
Oct 30 04:51:05.296721 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.296697    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eb760784-9728-48d0-9381-78fe27736f5f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd\" (UID: \"eb760784-9728-48d0-9381-78fe27736f5f\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd"
Oct 30 04:51:05.316296 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.316275    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trrrq\" (UniqueName: \"kubernetes.io/projected/eb760784-9728-48d0-9381-78fe27736f5f-kube-api-access-trrrq\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd\" (UID: \"eb760784-9728-48d0-9381-78fe27736f5f\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd"
Oct 30 04:51:05.422659 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.422575    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd"
Oct 30 04:51:05.727454 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:05.727429    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd"]
Oct 30 04:51:05.729485 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:51:05.729452    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb760784_9728_48d0_9381_78fe27736f5f.slice/crio-866aa0b17af7cd5d4811f3ad09340ab361ecd97a81918a3e0ebef8ee43bfb77e WatchSource:0}: Error finding container 866aa0b17af7cd5d4811f3ad09340ab361ecd97a81918a3e0ebef8ee43bfb77e: Status 404 returned error can't find the container with id 866aa0b17af7cd5d4811f3ad09340ab361ecd97a81918a3e0ebef8ee43bfb77e
Oct 30 04:51:06.282352 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:06.282313    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd" event={"ID":"eb760784-9728-48d0-9381-78fe27736f5f","Type":"ContainerStarted","Data":"bb3da43da5f846872da90840ddf4f4d5bf2d52e9f78ade3ce67b3b5ceb02c6af"}
Oct 30 04:51:06.282753 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:06.282373    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd" event={"ID":"eb760784-9728-48d0-9381-78fe27736f5f","Type":"ContainerStarted","Data":"866aa0b17af7cd5d4811f3ad09340ab361ecd97a81918a3e0ebef8ee43bfb77e"}
Oct 30 04:51:06.282753 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:06.282530    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd"
Oct 30 04:51:06.332524 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:06.332478    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd" podStartSLOduration=2.332467318 podStartE2EDuration="2.332467318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:51:06.329529298 +0000 UTC m=+3185.741924871" watchObservedRunningTime="2026-10-30 04:51:06.332467318 +0000 UTC m=+3185.744862892"
Oct 30 04:51:17.274402 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:17.274310    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274"]
Oct 30 04:51:17.286528 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:17.286500    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274"
Oct 30 04:51:17.345645 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:17.310678    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-fcf42a-dockercfg-28x6d\""
Oct 30 04:51:17.345645 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:17.311526    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-fcf42a\""
Oct 30 04:51:17.345645 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:17.311652    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv"]
Oct 30 04:51:17.345645 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:17.319768    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd"
Oct 30 04:51:17.345645 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:17.320595    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv"
Oct 30 04:51:17.345645 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:17.332399    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274"]
Oct 30 04:51:17.363897 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:17.363846    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c8bf07-6e78-4a49-a32c-302fe418d7cc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274\" (UID: \"b9c8bf07-6e78-4a49-a32c-302fe418d7cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274"
Oct 30 04:51:17.364292 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:17.364267    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8xq5\" (UniqueName: \"kubernetes.io/projected/b9c8bf07-6e78-4a49-a32c-302fe418d7cc-kube-api-access-m8xq5\") pod \"payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274\" (UID: \"b9c8bf07-6e78-4a49-a32c-302fe418d7cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274"
Oct 30 04:51:17.381652 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:17.381618    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv"]
Oct 30 04:51:17.465441 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:17.465403    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8xq5\" (UniqueName: \"kubernetes.io/projected/b9c8bf07-6e78-4a49-a32c-302fe418d7cc-kube-api-access-m8xq5\") pod \"payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274\" (UID: \"b9c8bf07-6e78-4a49-a32c-302fe418d7cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274"
Oct 30 04:51:17.465630 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:17.465483    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b71a43c4-1c1e-4e92-9a3d-68b6827efa7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv\" (UID: \"b71a43c4-1c1e-4e92-9a3d-68b6827efa7b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv"
Oct 30 04:51:17.465712 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:17.465674    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7qgw\" (UniqueName: \"kubernetes.io/projected/b71a43c4-1c1e-4e92-9a3d-68b6827efa7b-kube-api-access-j7qgw\") pod \"payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv\" (UID: \"b71a43c4-1c1e-4e92-9a3d-68b6827efa7b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv"
Oct 30 04:51:17.465918 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:17.465898    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c8bf07-6e78-4a49-a32c-302fe418d7cc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274\" (UID: \"b9c8bf07-6e78-4a49-a32c-302fe418d7cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274"
Oct 30 04:51:17.466602 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:17.466582    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c8bf07-6e78-4a49-a32c-302fe418d7cc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274\" (UID: \"b9c8bf07-6e78-4a49-a32c-302fe418d7cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274"
Oct 30 04:51:17.497619 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:17.497584    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8xq5\" (UniqueName: \"kubernetes.io/projected/b9c8bf07-6e78-4a49-a32c-302fe418d7cc-kube-api-access-m8xq5\") pod \"payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274\" (UID: \"b9c8bf07-6e78-4a49-a32c-302fe418d7cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274"
Oct 30 04:51:17.566896 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:17.566826    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b71a43c4-1c1e-4e92-9a3d-68b6827efa7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv\" (UID: \"b71a43c4-1c1e-4e92-9a3d-68b6827efa7b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv"
Oct 30 04:51:17.567051 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:17.566935    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7qgw\" (UniqueName: \"kubernetes.io/projected/b71a43c4-1c1e-4e92-9a3d-68b6827efa7b-kube-api-access-j7qgw\") pod \"payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv\" (UID: \"b71a43c4-1c1e-4e92-9a3d-68b6827efa7b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv"
Oct 30 04:51:17.567505 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:17.567482    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b71a43c4-1c1e-4e92-9a3d-68b6827efa7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv\" (UID: \"b71a43c4-1c1e-4e92-9a3d-68b6827efa7b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv"
Oct 30 04:51:17.590036 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:17.590006    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7qgw\" (UniqueName: \"kubernetes.io/projected/b71a43c4-1c1e-4e92-9a3d-68b6827efa7b-kube-api-access-j7qgw\") pod \"payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv\" (UID: \"b71a43c4-1c1e-4e92-9a3d-68b6827efa7b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv"
Oct 30 04:51:17.666193 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:17.666162    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274"
Oct 30 04:51:17.733567 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:17.732941    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv"
Oct 30 04:51:17.975207 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:17.974221    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274"]
Oct 30 04:51:17.975683 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:51:17.975653    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c8bf07_6e78_4a49_a32c_302fe418d7cc.slice/crio-c424a7c22263f644389026694d8d890d344a98ccf565fcbbd1feb72af9764414 WatchSource:0}: Error finding container c424a7c22263f644389026694d8d890d344a98ccf565fcbbd1feb72af9764414: Status 404 returned error can't find the container with id c424a7c22263f644389026694d8d890d344a98ccf565fcbbd1feb72af9764414
Oct 30 04:51:18.092669 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:18.092648    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv"]
Oct 30 04:51:18.094623 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:51:18.094600    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb71a43c4_1c1e_4e92_9a3d_68b6827efa7b.slice/crio-a4b4cf22b50e26b19761473b1af742e4671a5afdbcbd964c0d4db5c21f6b37dc WatchSource:0}: Error finding container a4b4cf22b50e26b19761473b1af742e4671a5afdbcbd964c0d4db5c21f6b37dc: Status 404 returned error can't find the container with id a4b4cf22b50e26b19761473b1af742e4671a5afdbcbd964c0d4db5c21f6b37dc
Oct 30 04:51:18.453915 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:18.453813    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv" event={"ID":"b71a43c4-1c1e-4e92-9a3d-68b6827efa7b","Type":"ContainerStarted","Data":"06053bfc780a89e82ba93a8cd7ff5a512721e4dba54a62cf376cd1365c543a5d"}
Oct 30 04:51:18.453915 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:18.453887    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv" event={"ID":"b71a43c4-1c1e-4e92-9a3d-68b6827efa7b","Type":"ContainerStarted","Data":"a4b4cf22b50e26b19761473b1af742e4671a5afdbcbd964c0d4db5c21f6b37dc"}
Oct 30 04:51:18.454466 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:18.453921    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv"
Oct 30 04:51:18.455495 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:18.455469    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274" event={"ID":"b9c8bf07-6e78-4a49-a32c-302fe418d7cc","Type":"ContainerStarted","Data":"036aba7c3b1892cda2ef41553fac984cc9b009805807ab79c9a8777407b18fbf"}
Oct 30 04:51:18.455636 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:18.455516    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274" event={"ID":"b9c8bf07-6e78-4a49-a32c-302fe418d7cc","Type":"ContainerStarted","Data":"c424a7c22263f644389026694d8d890d344a98ccf565fcbbd1feb72af9764414"}
Oct 30 04:51:18.455636 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:18.455607    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274"
Oct 30 04:51:18.505979 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:18.505917    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv" podStartSLOduration=1.505899446 podStartE2EDuration="1.505899446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:51:18.502406077 +0000 UTC m=+3197.914801652" watchObservedRunningTime="2026-10-30 04:51:18.505899446 +0000 UTC m=+3197.918295021"
Oct 30 04:51:18.561081 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:18.561037    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274" podStartSLOduration=1.561024008 podStartE2EDuration="1.561024008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:51:18.559142734 +0000 UTC m=+3197.971538310" watchObservedRunningTime="2026-10-30 04:51:18.561024008 +0000 UTC m=+3197.973419592"
Oct 30 04:51:25.610037 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:51:25.609998    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:51:29.483262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:29.483021    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274"
Oct 30 04:51:29.498774 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:29.484397    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv"
Oct 30 04:51:53.490124 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:53.490086    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:51:55.070529 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:55.070496    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw"]
Oct 30 04:51:55.107483 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:55.102861    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw"
Oct 30 04:51:55.136262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:55.125338    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-fcf42a-serving-cert\""
Oct 30 04:51:55.136262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:55.125391    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b7e4826-cc1f-401c-ad81-608f4234cf7f-metrics-certs\") pod \"maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw\" (UID: \"9b7e4826-cc1f-401c-ad81-608f4234cf7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw"
Oct 30 04:51:55.136262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:55.125605    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7e4826-cc1f-401c-ad81-608f4234cf7f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw\" (UID: \"9b7e4826-cc1f-401c-ad81-608f4234cf7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw"
Oct 30 04:51:55.136262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:55.125763    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc4v5\" (UniqueName: \"kubernetes.io/projected/9b7e4826-cc1f-401c-ad81-608f4234cf7f-kube-api-access-kc4v5\") pod \"maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw\" (UID: \"9b7e4826-cc1f-401c-ad81-608f4234cf7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw"
Oct 30 04:51:55.136262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:55.125844    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7e4826-cc1f-401c-ad81-608f4234cf7f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw\" (UID: \"9b7e4826-cc1f-401c-ad81-608f4234cf7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw"
Oct 30 04:51:55.136262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:55.130075    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw"]
Oct 30 04:51:55.227163 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:55.227134    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b7e4826-cc1f-401c-ad81-608f4234cf7f-metrics-certs\") pod \"maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw\" (UID: \"9b7e4826-cc1f-401c-ad81-608f4234cf7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw"
Oct 30 04:51:55.227340 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:55.227276    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7e4826-cc1f-401c-ad81-608f4234cf7f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw\" (UID: \"9b7e4826-cc1f-401c-ad81-608f4234cf7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw"
Oct 30 04:51:55.227407 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:55.227364    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc4v5\" (UniqueName: \"kubernetes.io/projected/9b7e4826-cc1f-401c-ad81-608f4234cf7f-kube-api-access-kc4v5\") pod \"maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw\" (UID: \"9b7e4826-cc1f-401c-ad81-608f4234cf7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw"
Oct 30 04:51:55.227453 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:55.227435    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7e4826-cc1f-401c-ad81-608f4234cf7f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw\" (UID: \"9b7e4826-cc1f-401c-ad81-608f4234cf7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw"
Oct 30 04:51:55.228105 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:55.228081    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7e4826-cc1f-401c-ad81-608f4234cf7f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw\" (UID: \"9b7e4826-cc1f-401c-ad81-608f4234cf7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw"
Oct 30 04:51:55.229615 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:55.229590    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b7e4826-cc1f-401c-ad81-608f4234cf7f-metrics-certs\") pod \"maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw\" (UID: \"9b7e4826-cc1f-401c-ad81-608f4234cf7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw"
Oct 30 04:51:55.229753 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:55.229733    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7e4826-cc1f-401c-ad81-608f4234cf7f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw\" (UID: \"9b7e4826-cc1f-401c-ad81-608f4234cf7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw"
Oct 30 04:51:55.249505 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:55.249474    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc4v5\" (UniqueName: \"kubernetes.io/projected/9b7e4826-cc1f-401c-ad81-608f4234cf7f-kube-api-access-kc4v5\") pod \"maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw\" (UID: \"9b7e4826-cc1f-401c-ad81-608f4234cf7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw"
Oct 30 04:51:55.492495 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:55.492457    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw"
Oct 30 04:51:55.613552 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:51:55.613494    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:51:55.800357 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:55.800332    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw"]
Oct 30 04:51:55.801887 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:51:55.801858    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7e4826_cc1f_401c_ad81_608f4234cf7f.slice/crio-dc254fb2780e0851a002e21a86d97e38b908a09b3a08ee4a90c1c98e67f5c2d1 WatchSource:0}: Error finding container dc254fb2780e0851a002e21a86d97e38b908a09b3a08ee4a90c1c98e67f5c2d1: Status 404 returned error can't find the container with id dc254fb2780e0851a002e21a86d97e38b908a09b3a08ee4a90c1c98e67f5c2d1
Oct 30 04:51:56.773906 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:56.773877    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw"
Oct 30 04:51:56.774445 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:56.773927    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw" event={"ID":"9b7e4826-cc1f-401c-ad81-608f4234cf7f","Type":"ContainerStarted","Data":"4d607ea1e08474103072523084939bc732dc5767ede2a3f8e9c89f0334b709c6"}
Oct 30 04:51:56.774445 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:56.773949    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw" event={"ID":"9b7e4826-cc1f-401c-ad81-608f4234cf7f","Type":"ContainerStarted","Data":"dc254fb2780e0851a002e21a86d97e38b908a09b3a08ee4a90c1c98e67f5c2d1"}
Oct 30 04:51:56.813070 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:51:56.813022    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw" podStartSLOduration=1.433408762 podStartE2EDuration="1.813005893s" totalImagesPullingTime="379.597131ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:51:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:51:56.812373246 +0000 UTC m=+3236.224768819" watchObservedRunningTime="2026-10-30 04:51:56.813005893 +0000 UTC m=+3236.225401467"
Oct 30 04:52:02.783649 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:02.783614    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw"
Oct 30 04:52:24.944463 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:24.944431    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"]
Oct 30 04:52:24.948813 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:24.948792    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:24.966705 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:24.966686    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-vwdzb\""
Oct 30 04:52:24.985114 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:24.985091    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"]
Oct 30 04:52:25.071435 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.071399    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.071620 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.071471    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.071620 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.071551    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.071620 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.071598    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.071762 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.071648    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2fx5\" (UniqueName: \"kubernetes.io/projected/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-kube-api-access-b2fx5\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.071808 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.071752    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.071850 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.071825    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.071890 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.071863    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.071890 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.071887    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.173459 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.173425    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2fx5\" (UniqueName: \"kubernetes.io/projected/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-kube-api-access-b2fx5\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.173643 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.173484    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.173643 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.173544    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.173643 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.173588    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.173643 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.173623    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.173897 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.173667    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.173897 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.173729    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.173897 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.173795    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.173897 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.173827    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.174113 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.174052    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.174173 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.174107    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.174408 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.174381    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.174521 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.174424    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.174584 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.174530    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.176109 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.176086    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.176328 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.176311    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.201470 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.201405    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.201784 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.201759    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2fx5\" (UniqueName: \"kubernetes.io/projected/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-kube-api-access-b2fx5\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rr9m8\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.262838 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.262815    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:25.686747 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:52:25.678448    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:52:25.686747 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.684715    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"]
Oct 30 04:52:25.690167 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:52:25.690132    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d33d6b0_7d09_4361_b56c_da12cddb3e8f.slice/crio-65607753b818187b80829c675a0030e88e98ba1aa734dd10c1e667ed8fa3ed58 WatchSource:0}: Error finding container 65607753b818187b80829c675a0030e88e98ba1aa734dd10c1e667ed8fa3ed58: Status 404 returned error can't find the container with id 65607753b818187b80829c675a0030e88e98ba1aa734dd10c1e667ed8fa3ed58
Oct 30 04:52:25.693525 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.693491    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:52:25.693664 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.693584    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:52:25.693664 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:25.693629    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:52:26.001899 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:26.001859    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8" event={"ID":"0d33d6b0-7d09-4361-b56c-da12cddb3e8f","Type":"ContainerStarted","Data":"7e8b3d2aff6f5b259cf1fc13b4d0045f1de6f40ad6a6645dd3add71b290fa72d"}
Oct 30 04:52:26.002328 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:26.001963    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8" event={"ID":"0d33d6b0-7d09-4361-b56c-da12cddb3e8f","Type":"ContainerStarted","Data":"65607753b818187b80829c675a0030e88e98ba1aa734dd10c1e667ed8fa3ed58"}
Oct 30 04:52:26.047768 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:26.047718    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8" podStartSLOduration=2.047704921 podStartE2EDuration="2.047704921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:26.04609205 +0000 UTC m=+3265.458487642" watchObservedRunningTime="2026-10-30 04:52:26.047704921 +0000 UTC m=+3265.460100494"
Oct 30 04:52:26.263964 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:26.263887    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:26.265155 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:26.265130    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rr9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 30 04:52:26.265309 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:26.265182    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8" podUID="0d33d6b0-7d09-4361-b56c-da12cddb3e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 30 04:52:27.276273 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:27.267629    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rr9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 30 04:52:27.276273 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:27.267697    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8" podUID="0d33d6b0-7d09-4361-b56c-da12cddb3e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 30 04:52:28.264115 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:28.264084    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rr9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 30 04:52:28.264310 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:28.264136    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8" podUID="0d33d6b0-7d09-4361-b56c-da12cddb3e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 30 04:52:29.264113 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:29.264080    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rr9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 30 04:52:29.264606 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:29.264133    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8" podUID="0d33d6b0-7d09-4361-b56c-da12cddb3e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 30 04:52:30.263397 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:30.263359    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rr9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 30 04:52:30.263584 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:30.263413    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8" podUID="0d33d6b0-7d09-4361-b56c-da12cddb3e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 30 04:52:31.266259 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:31.265562    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rr9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 30 04:52:31.266259 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:31.265627    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8" podUID="0d33d6b0-7d09-4361-b56c-da12cddb3e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 30 04:52:32.268261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:32.267685    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rr9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 30 04:52:32.268261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:32.267754    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8" podUID="0d33d6b0-7d09-4361-b56c-da12cddb3e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 30 04:52:33.263579 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:33.263546    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rr9m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 30 04:52:33.263759 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:33.263608    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8" podUID="0d33d6b0-7d09-4361-b56c-da12cddb3e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 30 04:52:34.665656 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:34.665622    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:34.666429 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:34.666034    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:34.666941 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:34.666925    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:52:35.372287 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:35.372235    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-45d763/e2e-usage-sim-258452-86c5bc89b4-k4lwj"]
Oct 30 04:52:35.376432 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:35.376409    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-45d763/e2e-usage-sim-258452-86c5bc89b4-k4lwj"
Oct 30 04:52:35.392423 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:35.392401    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-45d763\"/\"openshift-service-ca.crt\""
Oct 30 04:52:35.392551 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:35.392508    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-45d763\"/\"kube-root-ca.crt\""
Oct 30 04:52:35.392755 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:35.392739    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-45d763\"/\"e2e-usage-sim-258452-cert\""
Oct 30 04:52:35.397012 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:35.396990    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-45d763/e2e-usage-sim-258452-86c5bc89b4-k4lwj"]
Oct 30 04:52:35.403736 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:35.403703    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-45d763\"/\"default-dockercfg-l2dvf\""
Oct 30 04:52:35.418200 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:35.418170    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5n52\" (UniqueName: \"kubernetes.io/projected/177c06f9-15cd-4d6b-ab20-a3d7b3addb85-kube-api-access-h5n52\") pod \"e2e-usage-sim-258452-86c5bc89b4-k4lwj\" (UID: \"177c06f9-15cd-4d6b-ab20-a3d7b3addb85\") " pod="e2e-extmod-shr-user-45d763/e2e-usage-sim-258452-86c5bc89b4-k4lwj"
Oct 30 04:52:35.418351 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:35.418254    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/177c06f9-15cd-4d6b-ab20-a3d7b3addb85-tls\") pod \"e2e-usage-sim-258452-86c5bc89b4-k4lwj\" (UID: \"177c06f9-15cd-4d6b-ab20-a3d7b3addb85\") " pod="e2e-extmod-shr-user-45d763/e2e-usage-sim-258452-86c5bc89b4-k4lwj"
Oct 30 04:52:35.520006 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:35.519975    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/177c06f9-15cd-4d6b-ab20-a3d7b3addb85-tls\") pod \"e2e-usage-sim-258452-86c5bc89b4-k4lwj\" (UID: \"177c06f9-15cd-4d6b-ab20-a3d7b3addb85\") " pod="e2e-extmod-shr-user-45d763/e2e-usage-sim-258452-86c5bc89b4-k4lwj"
Oct 30 04:52:35.520186 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:35.520127    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5n52\" (UniqueName: \"kubernetes.io/projected/177c06f9-15cd-4d6b-ab20-a3d7b3addb85-kube-api-access-h5n52\") pod \"e2e-usage-sim-258452-86c5bc89b4-k4lwj\" (UID: \"177c06f9-15cd-4d6b-ab20-a3d7b3addb85\") " pod="e2e-extmod-shr-user-45d763/e2e-usage-sim-258452-86c5bc89b4-k4lwj"
Oct 30 04:52:35.522496 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:35.522471    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/177c06f9-15cd-4d6b-ab20-a3d7b3addb85-tls\") pod \"e2e-usage-sim-258452-86c5bc89b4-k4lwj\" (UID: \"177c06f9-15cd-4d6b-ab20-a3d7b3addb85\") " pod="e2e-extmod-shr-user-45d763/e2e-usage-sim-258452-86c5bc89b4-k4lwj"
Oct 30 04:52:35.542439 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:35.542407    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5n52\" (UniqueName: \"kubernetes.io/projected/177c06f9-15cd-4d6b-ab20-a3d7b3addb85-kube-api-access-h5n52\") pod \"e2e-usage-sim-258452-86c5bc89b4-k4lwj\" (UID: \"177c06f9-15cd-4d6b-ab20-a3d7b3addb85\") " pod="e2e-extmod-shr-user-45d763/e2e-usage-sim-258452-86c5bc89b4-k4lwj"
Oct 30 04:52:35.689927 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:35.689823    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-45d763/e2e-usage-sim-258452-86c5bc89b4-k4lwj"
Oct 30 04:52:36.057307 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:36.057284    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-45d763/e2e-usage-sim-258452-86c5bc89b4-k4lwj"]
Oct 30 04:52:36.059914 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:52:36.059882    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod177c06f9_15cd_4d6b_ab20_a3d7b3addb85.slice/crio-08662eba668aaf917cadd18aa0817f9544dfd71c9ce9a69c88f75a4aba7c3b68 WatchSource:0}: Error finding container 08662eba668aaf917cadd18aa0817f9544dfd71c9ce9a69c88f75a4aba7c3b68: Status 404 returned error can't find the container with id 08662eba668aaf917cadd18aa0817f9544dfd71c9ce9a69c88f75a4aba7c3b68
Oct 30 04:52:36.123688 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:36.123658    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-45d763/e2e-usage-sim-258452-86c5bc89b4-k4lwj" event={"ID":"177c06f9-15cd-4d6b-ab20-a3d7b3addb85","Type":"ContainerStarted","Data":"08662eba668aaf917cadd18aa0817f9544dfd71c9ce9a69c88f75a4aba7c3b68"}
Oct 30 04:52:37.132081 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:37.132044    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-45d763/e2e-usage-sim-258452-86c5bc89b4-k4lwj" event={"ID":"177c06f9-15cd-4d6b-ab20-a3d7b3addb85","Type":"ContainerStarted","Data":"63782fc9ec7d34a58994b5a0f33041f1634662d528d90234e136b3739fad2d58"}
Oct 30 04:52:37.132545 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:37.132092    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-45d763/e2e-usage-sim-258452-86c5bc89b4-k4lwj"
Oct 30 04:52:37.136885 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:37.136860    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-45d763/e2e-usage-sim-258452-86c5bc89b4-k4lwj"
Oct 30 04:52:37.171064 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:37.171018    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-45d763/e2e-usage-sim-258452-86c5bc89b4-k4lwj" podStartSLOduration=2.171006206 podStartE2EDuration="2.171006206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:37.168830721 +0000 UTC m=+3276.581226295" watchObservedRunningTime="2026-10-30 04:52:37.171006206 +0000 UTC m=+3276.583401778"
Oct 30 04:52:38.350700 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:38.350663    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd"]
Oct 30 04:52:40.041576 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.041539    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"]
Oct 30 04:52:40.048349 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.048321    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"]
Oct 30 04:52:40.056050 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.056018    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"]
Oct 30 04:52:40.056364 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.056346    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.058328 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.058285    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"]
Oct 30 04:52:40.059326 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.059142    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.105495 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.105470    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.105631 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.105502    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.105631 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.105531    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.105731 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.105629    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.105731 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.105674    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.105731 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.105724    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.105893 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.105837    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5gtr\" (UniqueName: \"kubernetes.io/projected/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-kube-api-access-x5gtr\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.105893 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.105872    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52058723-4696-41d4-aa59-e3ae0e81e9fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.106025 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.106010    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.106088 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.106038    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.106145 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.106087    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.106145 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.106129    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.106230 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.106183    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.106343 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.106234    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.106343 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.106312    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.106445 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.106429    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sl56\" (UniqueName: \"kubernetes.io/projected/52058723-4696-41d4-aa59-e3ae0e81e9fd-kube-api-access-9sl56\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.106505 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.106473    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.106557 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.106519    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.207823 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.207794    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.207995 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.207852    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5gtr\" (UniqueName: \"kubernetes.io/projected/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-kube-api-access-x5gtr\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.207995 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.207874    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52058723-4696-41d4-aa59-e3ae0e81e9fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.207995 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.207933    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.207995 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.207950    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.207995 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.207980    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.208316 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.208001    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.208316 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.208024    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.208316 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.208067    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.208316 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.208103    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.208316 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.208131    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sl56\" (UniqueName: \"kubernetes.io/projected/52058723-4696-41d4-aa59-e3ae0e81e9fd-kube-api-access-9sl56\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.208316 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.208170    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.208316 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.208218    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.208316 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.208299    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.208734 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.208331    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.208734 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.208374    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.208734 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.208435    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.208734 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.208467    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.208734 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.208480    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.208996 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.208796    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.208996 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.208817    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.208996 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.208858    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.208996 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.208976    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52058723-4696-41d4-aa59-e3ae0e81e9fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.209219 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.209046    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.209219 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.209066    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.209219 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.209120    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.209219 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.209157    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.209425 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.209401    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.210951 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.210920    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.211116 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.211095    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.211166 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.211126    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.211367 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.211351    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.230425 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.230395    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.230883 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.230862    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.230941 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.230903    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sl56\" (UniqueName: \"kubernetes.io/projected/52058723-4696-41d4-aa59-e3ae0e81e9fd-kube-api-access-9sl56\") pod \"maas-default-gateway-openshift-default-7f694df8bc-27xg6\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.230976 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.230963    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5gtr\" (UniqueName: \"kubernetes.io/projected/62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69-kube-api-access-x5gtr\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hwcsp\" (UID: \"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.392518 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.392439    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:40.405625 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.405600    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:40.794586 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.791757    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"]
Oct 30 04:52:40.794586 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.792790    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"]
Oct 30 04:52:40.883260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.878297    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:52:40.883260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.878399    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:52:40.883260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.878470    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:52:40.891058 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.891007    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:52:40.891294 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.891101    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:52:40.891294 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:40.891145    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:52:41.165993 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:41.165897    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp" event={"ID":"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69","Type":"ContainerStarted","Data":"128109673bb5cfb3381b72dfbe148550fe81cff4105a6c3d4693611829588940"}
Oct 30 04:52:41.165993 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:41.165965    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp" event={"ID":"62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69","Type":"ContainerStarted","Data":"556e6a375a2d82c019b2a8db97f0d863441f167c3cee17208e3e39939ef78a5b"}
Oct 30 04:52:41.167773 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:41.167745    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6" event={"ID":"52058723-4696-41d4-aa59-e3ae0e81e9fd","Type":"ContainerStarted","Data":"47e063b42e78c3ad0e72efcd8b00d71a5e2264b2a9f2656fe9b1460b5b43ae47"}
Oct 30 04:52:41.167923 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:41.167793    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6" event={"ID":"52058723-4696-41d4-aa59-e3ae0e81e9fd","Type":"ContainerStarted","Data":"a00d6ed1c3292acac316f0a128a991da2f24a8a0872c033be77065f2566396fd"}
Oct 30 04:52:41.218271 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:41.215212    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp" podStartSLOduration=2.215192148 podStartE2EDuration="2.215192148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:41.211569741 +0000 UTC m=+3280.623965315" watchObservedRunningTime="2026-10-30 04:52:41.215192148 +0000 UTC m=+3280.627587722"
Oct 30 04:52:41.279492 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:41.279319    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6" podStartSLOduration=2.279297831 podStartE2EDuration="2.279297831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:41.266952347 +0000 UTC m=+3280.679347922" watchObservedRunningTime="2026-10-30 04:52:41.279297831 +0000 UTC m=+3280.691693405"
Oct 30 04:52:41.392835 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:41.392799    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:41.394460 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:41.394430    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hwcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 04:52:41.394704 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:41.394670    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp" podUID="62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 04:52:41.406395 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:41.406369    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:41.407587 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:41.407566    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-27xg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 30 04:52:41.407673 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:41.407601    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6" podUID="52058723-4696-41d4-aa59-e3ae0e81e9fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 30 04:52:42.410261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:42.399624    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hwcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 04:52:42.410261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:42.399685    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp" podUID="62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 04:52:42.410261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:42.408626    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-27xg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 30 04:52:42.410261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:42.408678    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6" podUID="52058723-4696-41d4-aa59-e3ae0e81e9fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 30 04:52:43.351612 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:43.351520    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd" podUID="eb760784-9728-48d0-9381-78fe27736f5f" containerName="payload-processing" containerID="cri-o://bb3da43da5f846872da90840ddf4f4d5bf2d52e9f78ade3ce67b3b5ceb02c6af" gracePeriod=25
Oct 30 04:52:43.397018 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:43.396588    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hwcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 04:52:43.397018 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:43.396647    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp" podUID="62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 04:52:43.408609 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:43.408517    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-27xg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 30 04:52:43.408609 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:43.408563    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6" podUID="52058723-4696-41d4-aa59-e3ae0e81e9fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 30 04:52:43.915919 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:43.915562    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd"
Oct 30 04:52:43.995732 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:43.995696    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb760784-9728-48d0-9381-78fe27736f5f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eb760784-9728-48d0-9381-78fe27736f5f-routing-overlay-volume\") pod \"eb760784-9728-48d0-9381-78fe27736f5f\" (UID: \"eb760784-9728-48d0-9381-78fe27736f5f\") "
Oct 30 04:52:43.996016 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:43.995995    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb760784-9728-48d0-9381-78fe27736f5f-kube-api-access-trrrq\" (UniqueName: \"kubernetes.io/projected/eb760784-9728-48d0-9381-78fe27736f5f-kube-api-access-trrrq\") pod \"eb760784-9728-48d0-9381-78fe27736f5f\" (UID: \"eb760784-9728-48d0-9381-78fe27736f5f\") "
Oct 30 04:52:44.002273 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:43.996450    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb760784-9728-48d0-9381-78fe27736f5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb760784-9728-48d0-9381-78fe27736f5f-plugins-config-volume\") pod \"eb760784-9728-48d0-9381-78fe27736f5f\" (UID: \"eb760784-9728-48d0-9381-78fe27736f5f\") "
Oct 30 04:52:44.006554 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:44.006533    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb760784-9728-48d0-9381-78fe27736f5f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/eb760784-9728-48d0-9381-78fe27736f5f-provider-credentials\") pod \"eb760784-9728-48d0-9381-78fe27736f5f\" (UID: \"eb760784-9728-48d0-9381-78fe27736f5f\") "
Oct 30 04:52:44.024260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:43.997732    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb760784-9728-48d0-9381-78fe27736f5f-routing-overlay-volume" pod "eb760784-9728-48d0-9381-78fe27736f5f" (UID: "eb760784-9728-48d0-9381-78fe27736f5f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:52:44.024260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:44.006466    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb760784-9728-48d0-9381-78fe27736f5f-plugins-config-volume" pod "eb760784-9728-48d0-9381-78fe27736f5f" (UID: "eb760784-9728-48d0-9381-78fe27736f5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:52:44.027038 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:44.026992    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb760784-9728-48d0-9381-78fe27736f5f-provider-credentials" pod "eb760784-9728-48d0-9381-78fe27736f5f" (UID: "eb760784-9728-48d0-9381-78fe27736f5f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:52:44.028742 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:44.028716    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb760784-9728-48d0-9381-78fe27736f5f-kube-api-access-trrrq" pod "eb760784-9728-48d0-9381-78fe27736f5f" (UID: "eb760784-9728-48d0-9381-78fe27736f5f"). InnerVolumeSpecName "kube-api-access-trrrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:52:44.115289 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:44.114430    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb760784-9728-48d0-9381-78fe27736f5f-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:52:44.115289 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:44.114470    4874 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eb760784-9728-48d0-9381-78fe27736f5f-provider-credentials\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:52:44.115289 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:44.114484    4874 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eb760784-9728-48d0-9381-78fe27736f5f-routing-overlay-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:52:44.115289 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:44.114497    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trrrq\" (UniqueName: \"kubernetes.io/projected/eb760784-9728-48d0-9381-78fe27736f5f-kube-api-access-trrrq\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:52:44.239053 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:44.239013    4874 generic.go:356] "Generic (PLEG): container finished" podID="eb760784-9728-48d0-9381-78fe27736f5f" containerID="bb3da43da5f846872da90840ddf4f4d5bf2d52e9f78ade3ce67b3b5ceb02c6af" exitCode=0
Oct 30 04:52:44.239227 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:44.239060    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd"
Oct 30 04:52:44.239227 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:44.239092    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd" event={"ID":"eb760784-9728-48d0-9381-78fe27736f5f","Type":"ContainerDied","Data":"bb3da43da5f846872da90840ddf4f4d5bf2d52e9f78ade3ce67b3b5ceb02c6af"}
Oct 30 04:52:44.239227 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:44.239146    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd" event={"ID":"eb760784-9728-48d0-9381-78fe27736f5f","Type":"ContainerDied","Data":"866aa0b17af7cd5d4811f3ad09340ab361ecd97a81918a3e0ebef8ee43bfb77e"}
Oct 30 04:52:44.239227 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:44.239168    4874 scope.go:122] "RemoveContainer" containerID="bb3da43da5f846872da90840ddf4f4d5bf2d52e9f78ade3ce67b3b5ceb02c6af"
Oct 30 04:52:44.251003 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:44.250986    4874 scope.go:122] "RemoveContainer" containerID="bb3da43da5f846872da90840ddf4f4d5bf2d52e9f78ade3ce67b3b5ceb02c6af"
Oct 30 04:52:44.251267 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:52:44.251227    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb3da43da5f846872da90840ddf4f4d5bf2d52e9f78ade3ce67b3b5ceb02c6af\": container with ID starting with bb3da43da5f846872da90840ddf4f4d5bf2d52e9f78ade3ce67b3b5ceb02c6af not found: ID does not exist" containerID="bb3da43da5f846872da90840ddf4f4d5bf2d52e9f78ade3ce67b3b5ceb02c6af"
Oct 30 04:52:44.251353 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:44.251274    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3da43da5f846872da90840ddf4f4d5bf2d52e9f78ade3ce67b3b5ceb02c6af"} err="failed to get container status \"bb3da43da5f846872da90840ddf4f4d5bf2d52e9f78ade3ce67b3b5ceb02c6af\": rpc error: code = NotFound desc = could not find container \"bb3da43da5f846872da90840ddf4f4d5bf2d52e9f78ade3ce67b3b5ceb02c6af\": container with ID starting with bb3da43da5f846872da90840ddf4f4d5bf2d52e9f78ade3ce67b3b5ceb02c6af not found: ID does not exist"
Oct 30 04:52:44.318865 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:44.318822    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd"]
Oct 30 04:52:44.328137 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:44.328112    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-7bc8prgvd"]
Oct 30 04:52:44.394262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:44.393605    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hwcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 04:52:44.394262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:44.393668    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp" podUID="62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 04:52:44.414502 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:44.414342    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-27xg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 30 04:52:44.414502 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:44.414408    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6" podUID="52058723-4696-41d4-aa59-e3ae0e81e9fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 30 04:52:44.732753 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:44.732721    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb760784-9728-48d0-9381-78fe27736f5f" path="/var/lib/kubelet/pods/eb760784-9728-48d0-9381-78fe27736f5f/volumes"
Oct 30 04:52:45.396625 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:45.396587    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hwcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 04:52:45.397228 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:45.397195    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp" podUID="62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 04:52:45.409615 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:45.409581    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-27xg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 30 04:52:45.409843 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:45.409820    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6" podUID="52058723-4696-41d4-aa59-e3ae0e81e9fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 30 04:52:46.396272 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:46.396183    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hwcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 04:52:46.396532 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:46.396272    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp" podUID="62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 04:52:46.412264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:46.409510    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-27xg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 30 04:52:46.412264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:46.409564    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6" podUID="52058723-4696-41d4-aa59-e3ae0e81e9fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 30 04:52:47.395676 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:47.395639    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hwcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 04:52:47.396315 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:47.395911    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp" podUID="62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 04:52:47.411762 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:47.411638    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-27xg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 30 04:52:47.411762 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:47.411707    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6" podUID="52058723-4696-41d4-aa59-e3ae0e81e9fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 30 04:52:48.393105 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:48.393066    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hwcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 04:52:48.393531 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:48.393122    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp" podUID="62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 04:52:48.405968 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:48.405941    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-27xg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 30 04:52:48.406085 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:48.405978    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6" podUID="52058723-4696-41d4-aa59-e3ae0e81e9fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 30 04:52:49.976369 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:49.976327    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:49.977778 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:49.977755    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:49.978580 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:49.978562    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:52:50.007859 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:50.007825    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:50.008481 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:50.008448    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:50.009426 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:50.009405    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp"
Oct 30 04:52:52.720974 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:52.720943    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:52:55.165602 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.165541    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"]
Oct 30 04:52:55.166732 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.166706    4874 cpu_manager.go:414] "containerMap: removing container" podUID="eb760784-9728-48d0-9381-78fe27736f5f" containerName="payload-processing"
Oct 30 04:52:55.166868 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.166734    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb760784-9728-48d0-9381-78fe27736f5f" containerName="payload-processing"
Oct 30 04:52:55.166979 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.166891    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb760784-9728-48d0-9381-78fe27736f5f" containerName="payload-processing"
Oct 30 04:52:55.173783 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.173759    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.191522 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.191498    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5f1508-openshift-default-dockercfg-4jlkh\""
Oct 30 04:52:55.206455 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.206429    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"]
Oct 30 04:52:55.272492 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.272457    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-token\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.272676 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.272508    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-podinfo\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.272753 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.272711    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-data\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.272812 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.272795    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istiod-ca-cert\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.272865 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.272825    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2gp8\" (UniqueName: \"kubernetes.io/projected/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-kube-api-access-g2gp8\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.272923 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.272906    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-envoy\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.273001 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.272986    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-credential-socket\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.273056 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.273039    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-workload-socket\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.273096 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.273079    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-workload-certs\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.392261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.383717    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-credential-socket\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.392261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.383252    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-credential-socket\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.392261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.383841    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-workload-socket\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.392261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.383877    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-workload-certs\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.392261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.383934    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-token\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.392261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.383986    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-podinfo\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.392261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.384112    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-data\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.392261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.384169    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istiod-ca-cert\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.392261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.384196    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2gp8\" (UniqueName: \"kubernetes.io/projected/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-kube-api-access-g2gp8\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.392261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.384315    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-envoy\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.392261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.388090    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-workload-socket\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.392261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.388397    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-workload-certs\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.392261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.389009    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-data\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.392261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.389344    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istiod-ca-cert\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.418311 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.416558    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-podinfo\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.418311 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.416832    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-envoy\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.422260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.420741    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-token\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.443025 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.442960    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2gp8\" (UniqueName: \"kubernetes.io/projected/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-kube-api-access-g2gp8\") pod \"e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") " pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.508337 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.507351    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:55.624518 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:52:55.624460    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:52:55.910931 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.910853    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"]
Oct 30 04:52:55.914433 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:52:55.914401    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33dee17_9ac8_4ea0_b547_94b24e63b4bb.slice/crio-6ed4e92c927a28052af9bba06278e93a0575de8f0ec00425c5f9ea67ea22f7e7 WatchSource:0}: Error finding container 6ed4e92c927a28052af9bba06278e93a0575de8f0ec00425c5f9ea67ea22f7e7: Status 404 returned error can't find the container with id 6ed4e92c927a28052af9bba06278e93a0575de8f0ec00425c5f9ea67ea22f7e7
Oct 30 04:52:55.917459 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.917422    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:52:55.917550 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.917488    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:52:55.917550 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:55.917520    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:52:56.495471 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.495433    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7" event={"ID":"f33dee17-9ac8-4ea0-b547-94b24e63b4bb","Type":"ContainerStarted","Data":"fa28a29695912d6deadec13216d0bf916eacf092ffc9c8a670da8e5ed54b49a9"}
Oct 30 04:52:56.495866 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.495490    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7" event={"ID":"f33dee17-9ac8-4ea0-b547-94b24e63b4bb","Type":"ContainerStarted","Data":"6ed4e92c927a28052af9bba06278e93a0575de8f0ec00425c5f9ea67ea22f7e7"}
Oct 30 04:52:56.508998 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.508964    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:56.772649 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.772545    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7" podStartSLOduration=1.7725306459999999 podStartE2EDuration="1.772530646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:56.561300456 +0000 UTC m=+3295.973696046" watchObservedRunningTime="2026-10-30 04:52:56.772530646 +0000 UTC m=+3296.184926223"
Oct 30 04:52:56.774128 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.774102    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"]
Oct 30 04:52:56.778385 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.778363    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.805364 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.805337    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"]
Oct 30 04:52:56.835710 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.835678    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.835900 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.835877    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.835970 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.835918    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.836100 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.836079    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.836181 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.836121    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.836324 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.836298    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.836413 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.836339    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.836413 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.836403    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.836537 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.836439    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxm7m\" (UniqueName: \"kubernetes.io/projected/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-kube-api-access-lxm7m\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.836537 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.836441    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b4708a-openshift-default-dockercfg-t5rn6\""
Oct 30 04:52:56.938442 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.938408    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.938656 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.938458    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.938656 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.938569    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.938656 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.938593    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.938656 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.938654    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.938893 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.938689    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxm7m\" (UniqueName: \"kubernetes.io/projected/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-kube-api-access-lxm7m\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.938893 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.938729    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.938893 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.938853    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.938893 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.938869    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.938893 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.938881    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.939158 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.938915    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.939484 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.939452    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.939645 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.939524    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.939722 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.939700    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.941485 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.941462    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.941582 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.941491    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.961388 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.961367    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:56.961495 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:56.961443    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxm7m\" (UniqueName: \"kubernetes.io/projected/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-kube-api-access-lxm7m\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:57.094170 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:57.094090    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:57.429021 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:57.428997    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"]
Oct 30 04:52:57.433787 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:52:57.431385    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc7b1652_bcc3_412b_b416_3fecd1ec9b90.slice/crio-6ae7ffd41cb039ddbe1e2851774d8d49d09e0017807ebacd5d9e596421e8ff99 WatchSource:0}: Error finding container 6ae7ffd41cb039ddbe1e2851774d8d49d09e0017807ebacd5d9e596421e8ff99: Status 404 returned error can't find the container with id 6ae7ffd41cb039ddbe1e2851774d8d49d09e0017807ebacd5d9e596421e8ff99
Oct 30 04:52:57.435369 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:57.435336    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:52:57.435465 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:57.435407    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:52:57.435465 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:57.435436    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:52:57.503967 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:57.503939    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q" event={"ID":"dc7b1652-bcc3-412b-b416-3fecd1ec9b90","Type":"ContainerStarted","Data":"6ae7ffd41cb039ddbe1e2851774d8d49d09e0017807ebacd5d9e596421e8ff99"}
Oct 30 04:52:57.515747 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:57.515725    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:58.513088 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:58.513051    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q" event={"ID":"dc7b1652-bcc3-412b-b416-3fecd1ec9b90","Type":"ContainerStarted","Data":"6e4388c5246c264747eed1146ab8749a065952ad519e10b35a84ff8974de9355"}
Oct 30 04:52:58.513719 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:58.513694    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:58.514509 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:58.514490    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:52:58.628190 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:58.627694    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q" podStartSLOduration=2.627678087 podStartE2EDuration="2.627678087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:58.584769914 +0000 UTC m=+3297.997165495" watchObservedRunningTime="2026-10-30 04:52:58.627678087 +0000 UTC m=+3298.040073661"
Oct 30 04:52:59.094265 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:59.094199    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:52:59.096491 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:59.096065    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 30 04:52:59.096491 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:52:59.096113    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q" podUID="dc7b1652-bcc3-412b-b416-3fecd1ec9b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 30 04:53:00.095518 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:00.095477    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 30 04:53:00.095914 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:00.095536    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q" podUID="dc7b1652-bcc3-412b-b416-3fecd1ec9b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 30 04:53:01.094496 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:01.094464    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 30 04:53:01.094684 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:01.094516    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q" podUID="dc7b1652-bcc3-412b-b416-3fecd1ec9b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 30 04:53:01.100573 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:01.100548    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 04:53:01.137400 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:01.137378    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 04:53:02.094449 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:02.094413    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 30 04:53:02.094625 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:02.094469    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q" podUID="dc7b1652-bcc3-412b-b416-3fecd1ec9b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 30 04:53:03.108260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:03.107898    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 30 04:53:03.108260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:03.107959    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q" podUID="dc7b1652-bcc3-412b-b416-3fecd1ec9b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 30 04:53:04.094750 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:04.094717    4874 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 30 04:53:04.094926 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:04.094766    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q" podUID="dc7b1652-bcc3-412b-b416-3fecd1ec9b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 30 04:53:05.124297 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:05.119582    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:53:05.124297 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:05.121101    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:53:05.128570 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:05.128541    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 04:53:25.608838 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:53:25.608801    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:53:36.823234 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:36.823200    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:53:55.742685 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:53:55.742646    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:53:59.711972 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:59.711941    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg"]
Oct 30 04:53:59.717508 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:59.717484    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg"
Oct 30 04:53:59.732827 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:59.732800    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-134dad-serving-cert\""
Oct 30 04:53:59.749087 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:59.749060    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg"]
Oct 30 04:53:59.835842 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:59.835807    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd62bd73-c45b-465d-b3e5-93b0bf16570a-maas-api-tls\") pod \"maas-api-e2e-inf-a-134dad-548684db4c-7b5xg\" (UID: \"cd62bd73-c45b-465d-b3e5-93b0bf16570a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg"
Oct 30 04:53:59.835842 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:59.835847    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd62bd73-c45b-465d-b3e5-93b0bf16570a-metrics-certs\") pod \"maas-api-e2e-inf-a-134dad-548684db4c-7b5xg\" (UID: \"cd62bd73-c45b-465d-b3e5-93b0bf16570a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg"
Oct 30 04:53:59.836072 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:59.835988    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd62bd73-c45b-465d-b3e5-93b0bf16570a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-134dad-548684db4c-7b5xg\" (UID: \"cd62bd73-c45b-465d-b3e5-93b0bf16570a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg"
Oct 30 04:53:59.836226 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:59.836205    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwwxf\" (UniqueName: \"kubernetes.io/projected/cd62bd73-c45b-465d-b3e5-93b0bf16570a-kube-api-access-hwwxf\") pod \"maas-api-e2e-inf-a-134dad-548684db4c-7b5xg\" (UID: \"cd62bd73-c45b-465d-b3e5-93b0bf16570a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg"
Oct 30 04:53:59.940264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:59.938576    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd62bd73-c45b-465d-b3e5-93b0bf16570a-maas-api-tls\") pod \"maas-api-e2e-inf-a-134dad-548684db4c-7b5xg\" (UID: \"cd62bd73-c45b-465d-b3e5-93b0bf16570a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg"
Oct 30 04:53:59.940264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:59.938629    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd62bd73-c45b-465d-b3e5-93b0bf16570a-metrics-certs\") pod \"maas-api-e2e-inf-a-134dad-548684db4c-7b5xg\" (UID: \"cd62bd73-c45b-465d-b3e5-93b0bf16570a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg"
Oct 30 04:53:59.940264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:59.938696    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd62bd73-c45b-465d-b3e5-93b0bf16570a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-134dad-548684db4c-7b5xg\" (UID: \"cd62bd73-c45b-465d-b3e5-93b0bf16570a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg"
Oct 30 04:53:59.940264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:59.938819    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwwxf\" (UniqueName: \"kubernetes.io/projected/cd62bd73-c45b-465d-b3e5-93b0bf16570a-kube-api-access-hwwxf\") pod \"maas-api-e2e-inf-a-134dad-548684db4c-7b5xg\" (UID: \"cd62bd73-c45b-465d-b3e5-93b0bf16570a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg"
Oct 30 04:53:59.952103 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:59.952064    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd62bd73-c45b-465d-b3e5-93b0bf16570a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-134dad-548684db4c-7b5xg\" (UID: \"cd62bd73-c45b-465d-b3e5-93b0bf16570a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg"
Oct 30 04:53:59.977065 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:59.977031    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd62bd73-c45b-465d-b3e5-93b0bf16570a-maas-api-tls\") pod \"maas-api-e2e-inf-a-134dad-548684db4c-7b5xg\" (UID: \"cd62bd73-c45b-465d-b3e5-93b0bf16570a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg"
Oct 30 04:53:59.977966 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:59.977936    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd62bd73-c45b-465d-b3e5-93b0bf16570a-metrics-certs\") pod \"maas-api-e2e-inf-a-134dad-548684db4c-7b5xg\" (UID: \"cd62bd73-c45b-465d-b3e5-93b0bf16570a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg"
Oct 30 04:53:59.978841 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:53:59.978775    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwwxf\" (UniqueName: \"kubernetes.io/projected/cd62bd73-c45b-465d-b3e5-93b0bf16570a-kube-api-access-hwwxf\") pod \"maas-api-e2e-inf-a-134dad-548684db4c-7b5xg\" (UID: \"cd62bd73-c45b-465d-b3e5-93b0bf16570a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg"
Oct 30 04:54:00.031872 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:00.031842    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg"
Oct 30 04:54:00.388261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:00.387491    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg"]
Oct 30 04:54:00.395098 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:54:00.395068    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd62bd73_c45b_465d_b3e5_93b0bf16570a.slice/crio-2b0653a075b2acb5889196295aff89795e2152519eb8e8a7c6ea4b2985486b22 WatchSource:0}: Error finding container 2b0653a075b2acb5889196295aff89795e2152519eb8e8a7c6ea4b2985486b22: Status 404 returned error can't find the container with id 2b0653a075b2acb5889196295aff89795e2152519eb8e8a7c6ea4b2985486b22
Oct 30 04:54:01.162381 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:01.162303    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg" event={"ID":"cd62bd73-c45b-465d-b3e5-93b0bf16570a","Type":"ContainerStarted","Data":"d78f3469c270a2ebf1dac3a3cd0d40dc8b07cc0c2f63f4eefdaae711241f86d8"}
Oct 30 04:54:01.162381 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:01.162361    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg" event={"ID":"cd62bd73-c45b-465d-b3e5-93b0bf16570a","Type":"ContainerStarted","Data":"2b0653a075b2acb5889196295aff89795e2152519eb8e8a7c6ea4b2985486b22"}
Oct 30 04:54:01.162782 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:01.162511    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg"
Oct 30 04:54:01.205835 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:01.205781    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg" podStartSLOduration=1.776577873 podStartE2EDuration="2.205766681s" totalImagesPullingTime="429.188808ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:54:01.20355964 +0000 UTC m=+3360.615955211" watchObservedRunningTime="2026-10-30 04:54:01.205766681 +0000 UTC m=+3360.618162254"
Oct 30 04:54:06.206139 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:06.206109    4874 generic.go:356] "Generic (PLEG): container finished" podID="f33dee17-9ac8-4ea0-b547-94b24e63b4bb" containerID="fa28a29695912d6deadec13216d0bf916eacf092ffc9c8a670da8e5ed54b49a9" exitCode=0
Oct 30 04:54:06.206609 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:06.206183    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7" event={"ID":"f33dee17-9ac8-4ea0-b547-94b24e63b4bb","Type":"ContainerDied","Data":"fa28a29695912d6deadec13216d0bf916eacf092ffc9c8a670da8e5ed54b49a9"}
Oct 30 04:54:06.206609 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:06.206572    4874 scope.go:122] "RemoveContainer" containerID="fa28a29695912d6deadec13216d0bf916eacf092ffc9c8a670da8e5ed54b49a9"
Oct 30 04:54:06.208191 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:06.208166    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:54:06.208274 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:06.208230    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:54:06.208316 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:06.208287    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:54:07.191164 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:07.191131    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg"
Oct 30 04:54:07.214537 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:07.214505    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7" event={"ID":"f33dee17-9ac8-4ea0-b547-94b24e63b4bb","Type":"ContainerStarted","Data":"95d6e45010658a5cf414a1beb6294a30258481f8445b7172f192d6d93d6c424f"}
Oct 30 04:54:07.509185 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:07.509152    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:54:07.509394 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:07.509317    4874 patch_prober.go:29] interesting pod/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 30 04:54:07.509394 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:07.509371    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7" podUID="f33dee17-9ac8-4ea0-b547-94b24e63b4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 30 04:54:08.508935 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:08.508894    4874 patch_prober.go:29] interesting pod/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 30 04:54:08.509358 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:08.508958    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7" podUID="f33dee17-9ac8-4ea0-b547-94b24e63b4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 30 04:54:09.508805 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:09.508772    4874 patch_prober.go:29] interesting pod/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 30 04:54:09.508984 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:09.508822    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7" podUID="f33dee17-9ac8-4ea0-b547-94b24e63b4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 30 04:54:10.508824 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:10.508788    4874 patch_prober.go:29] interesting pod/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 30 04:54:10.509000 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:10.508852    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7" podUID="f33dee17-9ac8-4ea0-b547-94b24e63b4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 30 04:54:11.509170 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:11.509140    4874 patch_prober.go:29] interesting pod/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 30 04:54:11.509582 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:11.509193    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7" podUID="f33dee17-9ac8-4ea0-b547-94b24e63b4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 30 04:54:12.562591 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:12.562560    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:54:12.563448 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:12.563422    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:54:12.564370 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:12.564347    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 04:54:24.913311 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:24.906263    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:54:25.614983 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:54:25.614945    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:54:50.866417 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:50.866384    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"]
Oct 30 04:54:50.870802 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:50.870769    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:50.887698 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:50.887664    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-82bec5-openshift-default-dockercfg-sz8f7\""
Oct 30 04:54:50.901360 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:50.901323    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"]
Oct 30 04:54:51.036262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.018149    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-workload-socket\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.036262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.018320    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istiod-ca-cert\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.036262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.018394    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-envoy\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.036262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.018448    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-token\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.036262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.018478    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-data\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.036262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.018504    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-credential-socket\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.036262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.018546    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-podinfo\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.036262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.018691    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-workload-certs\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.036262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.018819    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6spg\" (UniqueName: \"kubernetes.io/projected/b35dc9fb-6f3c-4425-8360-6f52c94f1909-kube-api-access-c6spg\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.126350 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.126256    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-token\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.127068 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.127039    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-data\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.127209 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.127080    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-credential-socket\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.127209 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.127125    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-podinfo\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.127387 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.127221    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-workload-certs\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.127387 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.127353    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6spg\" (UniqueName: \"kubernetes.io/projected/b35dc9fb-6f3c-4425-8360-6f52c94f1909-kube-api-access-c6spg\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.127494 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.127437    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-workload-socket\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.127551 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.127521    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istiod-ca-cert\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.127609 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.127589    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-envoy\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.128120 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.128094    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-workload-certs\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.128295 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.128110    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-workload-socket\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.128648 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.128628    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-data\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.128785 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.128763    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istiod-ca-cert\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.129118 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.129097    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-credential-socket\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.156827 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.156797    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-podinfo\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.164565 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.164530    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-envoy\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.165209 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.165188    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-token\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.172339 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.165579    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6spg\" (UniqueName: \"kubernetes.io/projected/b35dc9fb-6f3c-4425-8360-6f52c94f1909-kube-api-access-c6spg\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.197618 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.197590    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:51.727264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.720644    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"]
Oct 30 04:54:51.844764 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.837988    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:54:51.844764 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.838090    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:54:51.844764 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:51.838141    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:54:52.737769 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:52.737738    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8" event={"ID":"b35dc9fb-6f3c-4425-8360-6f52c94f1909","Type":"ContainerStarted","Data":"4d7a9936c605f39601376b3421d7f3911c932dcd010bb07b11d456b28eceb6bd"}
Oct 30 04:54:52.738169 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:52.737788    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8" event={"ID":"b35dc9fb-6f3c-4425-8360-6f52c94f1909","Type":"ContainerStarted","Data":"27241d7171c80562c248a453b74a51a448b12ceeb3abba26bb482c9118e62961"}
Oct 30 04:54:52.797357 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:52.797308    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8" podStartSLOduration=2.797294274 podStartE2EDuration="2.797294274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:54:52.793699376 +0000 UTC m=+3412.206094949" watchObservedRunningTime="2026-10-30 04:54:52.797294274 +0000 UTC m=+3412.209689847"
Oct 30 04:54:53.198718 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:53.198647    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:53.204699 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:53.204677    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:53.744159 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:53.744126    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:53.745089 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:54:53.745066    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:54:55.612561 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:54:55.612522    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:55:16.149261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:16.135867    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:55:25.611406 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:55:25.611368    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:55:41.771062 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:41.771023    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz"]
Oct 30 04:55:41.775501 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:41.775476    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz"
Oct 30 04:55:41.791534 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:41.791509    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-82bec5-serving-cert\""
Oct 30 04:55:41.808120 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:41.808093    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz"]
Oct 30 04:55:41.850435 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:41.850398    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b1ce86c-0464-4511-bed6-acefb0d3b841-maas-api-tls\") pod \"maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz\" (UID: \"2b1ce86c-0464-4511-bed6-acefb0d3b841\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz"
Oct 30 04:55:41.850622 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:41.850445    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b1ce86c-0464-4511-bed6-acefb0d3b841-openshift-service-ca\") pod \"maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz\" (UID: \"2b1ce86c-0464-4511-bed6-acefb0d3b841\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz"
Oct 30 04:55:41.850688 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:41.850629    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g79sc\" (UniqueName: \"kubernetes.io/projected/2b1ce86c-0464-4511-bed6-acefb0d3b841-kube-api-access-g79sc\") pod \"maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz\" (UID: \"2b1ce86c-0464-4511-bed6-acefb0d3b841\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz"
Oct 30 04:55:41.850853 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:41.850835    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b1ce86c-0464-4511-bed6-acefb0d3b841-metrics-certs\") pod \"maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz\" (UID: \"2b1ce86c-0464-4511-bed6-acefb0d3b841\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz"
Oct 30 04:55:41.976270 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:41.963821    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b1ce86c-0464-4511-bed6-acefb0d3b841-maas-api-tls\") pod \"maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz\" (UID: \"2b1ce86c-0464-4511-bed6-acefb0d3b841\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz"
Oct 30 04:55:41.976604 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:41.976570    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b1ce86c-0464-4511-bed6-acefb0d3b841-openshift-service-ca\") pod \"maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz\" (UID: \"2b1ce86c-0464-4511-bed6-acefb0d3b841\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz"
Oct 30 04:55:41.976890 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:41.976863    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g79sc\" (UniqueName: \"kubernetes.io/projected/2b1ce86c-0464-4511-bed6-acefb0d3b841-kube-api-access-g79sc\") pod \"maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz\" (UID: \"2b1ce86c-0464-4511-bed6-acefb0d3b841\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz"
Oct 30 04:55:41.977171 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:41.977150    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b1ce86c-0464-4511-bed6-acefb0d3b841-metrics-certs\") pod \"maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz\" (UID: \"2b1ce86c-0464-4511-bed6-acefb0d3b841\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz"
Oct 30 04:55:41.991223 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:55:41.974432    4874 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-serving-cert: secret "maas-api-e2e-inf-b-82bec5-serving-cert" not found
Oct 30 04:55:41.991493 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:55:41.991475    4874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b1ce86c-0464-4511-bed6-acefb0d3b841-maas-api-tls podName:2b1ce86c-0464-4511-bed6-acefb0d3b841 nodeName:}" failed. No retries permitted until 2026-10-30 04:55:42.4914484 +0000 UTC m=+3461.903843974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2b1ce86c-0464-4511-bed6-acefb0d3b841-maas-api-tls") pod "maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz" (UID: "2b1ce86c-0464-4511-bed6-acefb0d3b841") : secret "maas-api-e2e-inf-b-82bec5-serving-cert" not found
Oct 30 04:55:41.991937 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:41.978129    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b1ce86c-0464-4511-bed6-acefb0d3b841-openshift-service-ca\") pod \"maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz\" (UID: \"2b1ce86c-0464-4511-bed6-acefb0d3b841\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz"
Oct 30 04:55:42.003809 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:42.003129    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g79sc\" (UniqueName: \"kubernetes.io/projected/2b1ce86c-0464-4511-bed6-acefb0d3b841-kube-api-access-g79sc\") pod \"maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz\" (UID: \"2b1ce86c-0464-4511-bed6-acefb0d3b841\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz"
Oct 30 04:55:42.015426 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:42.015389    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b1ce86c-0464-4511-bed6-acefb0d3b841-metrics-certs\") pod \"maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz\" (UID: \"2b1ce86c-0464-4511-bed6-acefb0d3b841\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz"
Oct 30 04:55:42.496685 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:42.496644    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b1ce86c-0464-4511-bed6-acefb0d3b841-maas-api-tls\") pod \"maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz\" (UID: \"2b1ce86c-0464-4511-bed6-acefb0d3b841\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz"
Oct 30 04:55:42.502950 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:42.502918    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b1ce86c-0464-4511-bed6-acefb0d3b841-maas-api-tls\") pod \"maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz\" (UID: \"2b1ce86c-0464-4511-bed6-acefb0d3b841\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz"
Oct 30 04:55:42.689949 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:42.689907    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz"
Oct 30 04:55:43.080714 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:55:43.080685    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1ce86c_0464_4511_bed6_acefb0d3b841.slice/crio-e18858371dfe5701fb4d27e2ab2d024a514f2985fab05ceda2461c43256c15ae WatchSource:0}: Error finding container e18858371dfe5701fb4d27e2ab2d024a514f2985fab05ceda2461c43256c15ae: Status 404 returned error can't find the container with id e18858371dfe5701fb4d27e2ab2d024a514f2985fab05ceda2461c43256c15ae
Oct 30 04:55:43.081209 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:43.080709    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz"]
Oct 30 04:55:43.084955 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:43.084937    4874 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 04:55:43.111609 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:43.111582    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz" event={"ID":"2b1ce86c-0464-4511-bed6-acefb0d3b841","Type":"ContainerStarted","Data":"e18858371dfe5701fb4d27e2ab2d024a514f2985fab05ceda2461c43256c15ae"}
Oct 30 04:55:45.126622 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:45.126583    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz" event={"ID":"2b1ce86c-0464-4511-bed6-acefb0d3b841","Type":"ContainerStarted","Data":"0dd75d783a8524acfad44c4211bc04fd0c85cbad987b6d80c080cbae5e55e785"}
Oct 30 04:55:45.126992 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:45.126651    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz"
Oct 30 04:55:45.171439 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:45.171384    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz" podStartSLOduration=2.975969117 podStartE2EDuration="4.171371187s" totalImagesPullingTime="1.19540207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:55:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:55:45.168504928 +0000 UTC m=+3464.580900502" watchObservedRunningTime="2026-10-30 04:55:45.171371187 +0000 UTC m=+3464.583766765"
Oct 30 04:55:51.139675 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:55:51.139646    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz"
Oct 30 04:55:55.610182 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:55:55.610146    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:56:06.434270 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:06.433818    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:56:12.910261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:12.907522    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-45d763/e2e-usage-sim-258452-86c5bc89b4-k4lwj"]
Oct 30 04:56:13.374167 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:13.374105    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-45d763/e2e-usage-sim-258452-86c5bc89b4-k4lwj" podUID="177c06f9-15cd-4d6b-ab20-a3d7b3addb85" containerName="sim" containerID="cri-o://63782fc9ec7d34a58994b5a0f33041f1634662d528d90234e136b3739fad2d58" gracePeriod=30
Oct 30 04:56:13.766086 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:13.766059    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-45d763/e2e-usage-sim-258452-86c5bc89b4-k4lwj"
Oct 30 04:56:13.801335 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:13.801305    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/177c06f9-15cd-4d6b-ab20-a3d7b3addb85-tls\" (UniqueName: \"kubernetes.io/secret/177c06f9-15cd-4d6b-ab20-a3d7b3addb85-tls\") pod \"177c06f9-15cd-4d6b-ab20-a3d7b3addb85\" (UID: \"177c06f9-15cd-4d6b-ab20-a3d7b3addb85\") "
Oct 30 04:56:13.801488 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:13.801350    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/177c06f9-15cd-4d6b-ab20-a3d7b3addb85-kube-api-access-h5n52\" (UniqueName: \"kubernetes.io/projected/177c06f9-15cd-4d6b-ab20-a3d7b3addb85-kube-api-access-h5n52\") pod \"177c06f9-15cd-4d6b-ab20-a3d7b3addb85\" (UID: \"177c06f9-15cd-4d6b-ab20-a3d7b3addb85\") "
Oct 30 04:56:13.803636 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:13.803604    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/177c06f9-15cd-4d6b-ab20-a3d7b3addb85-tls" pod "177c06f9-15cd-4d6b-ab20-a3d7b3addb85" (UID: "177c06f9-15cd-4d6b-ab20-a3d7b3addb85"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:56:13.803744 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:13.803661    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/177c06f9-15cd-4d6b-ab20-a3d7b3addb85-kube-api-access-h5n52" pod "177c06f9-15cd-4d6b-ab20-a3d7b3addb85" (UID: "177c06f9-15cd-4d6b-ab20-a3d7b3addb85"). InnerVolumeSpecName "kube-api-access-h5n52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:56:13.903333 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:13.903252    4874 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/177c06f9-15cd-4d6b-ab20-a3d7b3addb85-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:56:13.903333 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:13.903282    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5n52\" (UniqueName: \"kubernetes.io/projected/177c06f9-15cd-4d6b-ab20-a3d7b3addb85-kube-api-access-h5n52\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:56:14.466269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:14.463173    4874 generic.go:356] "Generic (PLEG): container finished" podID="177c06f9-15cd-4d6b-ab20-a3d7b3addb85" containerID="63782fc9ec7d34a58994b5a0f33041f1634662d528d90234e136b3739fad2d58" exitCode=0
Oct 30 04:56:14.466269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:14.463379    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-45d763/e2e-usage-sim-258452-86c5bc89b4-k4lwj" event={"ID":"177c06f9-15cd-4d6b-ab20-a3d7b3addb85","Type":"ContainerDied","Data":"63782fc9ec7d34a58994b5a0f33041f1634662d528d90234e136b3739fad2d58"}
Oct 30 04:56:14.466269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:14.463422    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-45d763/e2e-usage-sim-258452-86c5bc89b4-k4lwj" event={"ID":"177c06f9-15cd-4d6b-ab20-a3d7b3addb85","Type":"ContainerDied","Data":"08662eba668aaf917cadd18aa0817f9544dfd71c9ce9a69c88f75a4aba7c3b68"}
Oct 30 04:56:14.466269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:14.463446    4874 scope.go:122] "RemoveContainer" containerID="63782fc9ec7d34a58994b5a0f33041f1634662d528d90234e136b3739fad2d58"
Oct 30 04:56:14.466269 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:14.463601    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-45d763/e2e-usage-sim-258452-86c5bc89b4-k4lwj"
Oct 30 04:56:14.528013 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:14.527212    4874 scope.go:122] "RemoveContainer" containerID="63782fc9ec7d34a58994b5a0f33041f1634662d528d90234e136b3739fad2d58"
Oct 30 04:56:14.528013 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:56:14.527749    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63782fc9ec7d34a58994b5a0f33041f1634662d528d90234e136b3739fad2d58\": container with ID starting with 63782fc9ec7d34a58994b5a0f33041f1634662d528d90234e136b3739fad2d58 not found: ID does not exist" containerID="63782fc9ec7d34a58994b5a0f33041f1634662d528d90234e136b3739fad2d58"
Oct 30 04:56:14.528013 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:14.527792    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63782fc9ec7d34a58994b5a0f33041f1634662d528d90234e136b3739fad2d58"} err="failed to get container status \"63782fc9ec7d34a58994b5a0f33041f1634662d528d90234e136b3739fad2d58\": rpc error: code = NotFound desc = could not find container \"63782fc9ec7d34a58994b5a0f33041f1634662d528d90234e136b3739fad2d58\": container with ID starting with 63782fc9ec7d34a58994b5a0f33041f1634662d528d90234e136b3739fad2d58 not found: ID does not exist"
Oct 30 04:56:14.551387 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:14.551353    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-45d763/e2e-usage-sim-258452-86c5bc89b4-k4lwj"]
Oct 30 04:56:14.556394 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:14.556369    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-45d763/e2e-usage-sim-258452-86c5bc89b4-k4lwj"]
Oct 30 04:56:14.734298 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:14.734225    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="177c06f9-15cd-4d6b-ab20-a3d7b3addb85" path="/var/lib/kubelet/pods/177c06f9-15cd-4d6b-ab20-a3d7b3addb85/volumes"
Oct 30 04:56:18.512005 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:18.511972    4874 generic.go:356] "Generic (PLEG): container finished" podID="b35dc9fb-6f3c-4425-8360-6f52c94f1909" containerID="4d7a9936c605f39601376b3421d7f3911c932dcd010bb07b11d456b28eceb6bd" exitCode=0
Oct 30 04:56:18.512474 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:18.512033    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8" event={"ID":"b35dc9fb-6f3c-4425-8360-6f52c94f1909","Type":"ContainerDied","Data":"4d7a9936c605f39601376b3421d7f3911c932dcd010bb07b11d456b28eceb6bd"}
Oct 30 04:56:18.512474 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:18.512436    4874 scope.go:122] "RemoveContainer" containerID="4d7a9936c605f39601376b3421d7f3911c932dcd010bb07b11d456b28eceb6bd"
Oct 30 04:56:18.514087 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:18.514054    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:56:18.514158 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:18.514129    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:56:18.514194 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:18.514169    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:56:19.520755 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:19.520724    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8" event={"ID":"b35dc9fb-6f3c-4425-8360-6f52c94f1909","Type":"ContainerStarted","Data":"3d74768d638f2a73fd7dc15ab7f55f9568afa3c3c80e5b48cb887b8269fd133c"}
Oct 30 04:56:20.197812 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:20.197773    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:56:20.199860 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:20.199824    4874 patch_prober.go:29] interesting pod/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 04:56:20.200015 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:20.199889    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8" podUID="b35dc9fb-6f3c-4425-8360-6f52c94f1909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 04:56:21.198387 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:21.198354    4874 patch_prober.go:29] interesting pod/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 04:56:21.198799 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:21.198404    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8" podUID="b35dc9fb-6f3c-4425-8360-6f52c94f1909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 04:56:22.198781 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:22.198742    4874 patch_prober.go:29] interesting pod/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 04:56:22.199200 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:22.198797    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8" podUID="b35dc9fb-6f3c-4425-8360-6f52c94f1909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 04:56:23.198520 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:23.198461    4874 patch_prober.go:29] interesting pod/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 04:56:23.198706 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:23.198541    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8" podUID="b35dc9fb-6f3c-4425-8360-6f52c94f1909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 04:56:23.744004 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:23.743973    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:56:24.198213 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:24.198126    4874 patch_prober.go:29] interesting pod/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 04:56:24.198213 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:24.198198    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8" podUID="b35dc9fb-6f3c-4425-8360-6f52c94f1909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 04:56:25.203533 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:25.203501    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:56:25.204694 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:25.204670    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 04:56:25.610746 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:56:25.610716    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:56:34.243927 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.243272    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"]
Oct 30 04:56:34.244462 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.244313    4874 cpu_manager.go:414] "containerMap: removing container" podUID="177c06f9-15cd-4d6b-ab20-a3d7b3addb85" containerName="sim"
Oct 30 04:56:34.244462 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.244332    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="177c06f9-15cd-4d6b-ab20-a3d7b3addb85" containerName="sim"
Oct 30 04:56:34.244583 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.244472    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="177c06f9-15cd-4d6b-ab20-a3d7b3addb85" containerName="sim"
Oct 30 04:56:34.248502 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.248472    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 04:56:34.267044 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.267018    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f9cd17\"/\"openshift-service-ca.crt\""
Oct 30 04:56:34.269758 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.269735    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"]
Oct 30 04:56:34.273806 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.273785    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f9cd17\"/\"default-dockercfg-h7znq\""
Oct 30 04:56:34.273806 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.273785    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f9cd17\"/\"test-model-f9cd17-kserve-self-signed-certs\""
Oct 30 04:56:34.274085 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.274066    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f9cd17\"/\"kube-root-ca.crt\""
Oct 30 04:56:34.330058 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.330027    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-home\") pod \"test-model-f9cd17-kserve-865d955fc9-mbgkr\" (UID: \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\") " pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 04:56:34.330199 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.330108    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-model-cache\") pod \"test-model-f9cd17-kserve-865d955fc9-mbgkr\" (UID: \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\") " pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 04:56:34.330199 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.330184    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-dshm\") pod \"test-model-f9cd17-kserve-865d955fc9-mbgkr\" (UID: \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\") " pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 04:56:34.330335 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.330217    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-tmp-dir\") pod \"test-model-f9cd17-kserve-865d955fc9-mbgkr\" (UID: \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\") " pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 04:56:34.330418 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.330404    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftw6c\" (UniqueName: \"kubernetes.io/projected/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-kube-api-access-ftw6c\") pod \"test-model-f9cd17-kserve-865d955fc9-mbgkr\" (UID: \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\") " pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 04:56:34.330464 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.330451    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-tls-certs\") pod \"test-model-f9cd17-kserve-865d955fc9-mbgkr\" (UID: \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\") " pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 04:56:34.431489 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.431461    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftw6c\" (UniqueName: \"kubernetes.io/projected/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-kube-api-access-ftw6c\") pod \"test-model-f9cd17-kserve-865d955fc9-mbgkr\" (UID: \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\") " pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 04:56:34.431617 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.431520    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-tls-certs\") pod \"test-model-f9cd17-kserve-865d955fc9-mbgkr\" (UID: \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\") " pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 04:56:34.431617 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.431588    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-home\") pod \"test-model-f9cd17-kserve-865d955fc9-mbgkr\" (UID: \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\") " pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 04:56:34.431696 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.431639    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-model-cache\") pod \"test-model-f9cd17-kserve-865d955fc9-mbgkr\" (UID: \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\") " pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 04:56:34.431696 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.431675    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-dshm\") pod \"test-model-f9cd17-kserve-865d955fc9-mbgkr\" (UID: \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\") " pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 04:56:34.431774 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.431705    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-tmp-dir\") pod \"test-model-f9cd17-kserve-865d955fc9-mbgkr\" (UID: \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\") " pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 04:56:34.432125 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.432100    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-model-cache\") pod \"test-model-f9cd17-kserve-865d955fc9-mbgkr\" (UID: \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\") " pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 04:56:34.432254 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.432126    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-home\") pod \"test-model-f9cd17-kserve-865d955fc9-mbgkr\" (UID: \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\") " pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 04:56:34.432254 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.432161    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-tmp-dir\") pod \"test-model-f9cd17-kserve-865d955fc9-mbgkr\" (UID: \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\") " pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 04:56:34.433846 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.433821    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-dshm\") pod \"test-model-f9cd17-kserve-865d955fc9-mbgkr\" (UID: \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\") " pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 04:56:34.434047 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.434031    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-tls-certs\") pod \"test-model-f9cd17-kserve-865d955fc9-mbgkr\" (UID: \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\") " pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 04:56:34.453913 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.453889    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftw6c\" (UniqueName: \"kubernetes.io/projected/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-kube-api-access-ftw6c\") pod \"test-model-f9cd17-kserve-865d955fc9-mbgkr\" (UID: \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\") " pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 04:56:34.565014 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.564587    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 04:56:34.973764 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:34.973673    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"]
Oct 30 04:56:34.983501 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:56:34.983459    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1e88b87_85fa_4cce_ad5d_927bcddc23e9.slice/crio-f00c9e4ff0f867b5f0c1bf2c93a9e592cb41d1f1dbcf7269c7077db8632d4c26 WatchSource:0}: Error finding container f00c9e4ff0f867b5f0c1bf2c93a9e592cb41d1f1dbcf7269c7077db8632d4c26: Status 404 returned error can't find the container with id f00c9e4ff0f867b5f0c1bf2c93a9e592cb41d1f1dbcf7269c7077db8632d4c26
Oct 30 04:56:35.719787 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:35.719753    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr" event={"ID":"f1e88b87-85fa-4cce-ad5d-927bcddc23e9","Type":"ContainerStarted","Data":"eda64d8ac7985380372c5fade0a90c6ba99fbaf13ea22c80073405de2e21e481"}
Oct 30 04:56:35.719787 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:35.719792    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr" event={"ID":"f1e88b87-85fa-4cce-ad5d-927bcddc23e9","Type":"ContainerStarted","Data":"f00c9e4ff0f867b5f0c1bf2c93a9e592cb41d1f1dbcf7269c7077db8632d4c26"}
Oct 30 04:56:35.762674 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:35.762619    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr" podStartSLOduration=1.762601016 podStartE2EDuration="1.762601016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:35.758991506 +0000 UTC m=+3515.171387079" watchObservedRunningTime="2026-10-30 04:56:35.762601016 +0000 UTC m=+3515.174996591"
Oct 30 04:56:36.565186 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:36.565145    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 04:56:40.722270 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:40.721726    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"]
Oct 30 04:56:40.840444 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:40.839970    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 04:56:40.860409 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:40.858909    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-134dad\"/\"openshift-service-ca.crt\""
Oct 30 04:56:40.869415 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:40.862219    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-home\") pod \"test-model-134dad-kserve-88444b5bf-n8nkf\" (UID: \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\") " pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 04:56:40.869415 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:40.862393    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b4d251c-21f8-4b72-99c2-3efe08d34dda-tls-certs\") pod \"test-model-134dad-kserve-88444b5bf-n8nkf\" (UID: \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\") " pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 04:56:40.869415 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:40.862474    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-model-cache\") pod \"test-model-134dad-kserve-88444b5bf-n8nkf\" (UID: \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\") " pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 04:56:40.869415 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:40.862545    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-dshm\") pod \"test-model-134dad-kserve-88444b5bf-n8nkf\" (UID: \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\") " pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 04:56:40.869415 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:40.862578    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-tmp-dir\") pod \"test-model-134dad-kserve-88444b5bf-n8nkf\" (UID: \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\") " pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 04:56:40.869415 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:40.862650    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4vs8\" (UniqueName: \"kubernetes.io/projected/0b4d251c-21f8-4b72-99c2-3efe08d34dda-kube-api-access-x4vs8\") pod \"test-model-134dad-kserve-88444b5bf-n8nkf\" (UID: \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\") " pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 04:56:40.869415 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:40.863666    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"]
Oct 30 04:56:40.877270 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:40.870065    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-134dad\"/\"default-dockercfg-x5b67\""
Oct 30 04:56:40.958536 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:40.958500    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-134dad\"/\"kube-root-ca.crt\""
Oct 30 04:56:40.959421 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:40.959397    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-134dad\"/\"test-model-134dad-kserve-self-signed-certs\""
Oct 30 04:56:40.964349 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:40.964316    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-home\") pod \"test-model-134dad-kserve-88444b5bf-n8nkf\" (UID: \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\") " pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 04:56:40.964486 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:40.964424    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b4d251c-21f8-4b72-99c2-3efe08d34dda-tls-certs\") pod \"test-model-134dad-kserve-88444b5bf-n8nkf\" (UID: \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\") " pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 04:56:40.965130 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:40.964632    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-model-cache\") pod \"test-model-134dad-kserve-88444b5bf-n8nkf\" (UID: \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\") " pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 04:56:40.965130 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:40.964743    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-dshm\") pod \"test-model-134dad-kserve-88444b5bf-n8nkf\" (UID: \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\") " pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 04:56:40.965130 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:40.964772    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-tmp-dir\") pod \"test-model-134dad-kserve-88444b5bf-n8nkf\" (UID: \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\") " pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 04:56:40.965130 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:40.964866    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4vs8\" (UniqueName: \"kubernetes.io/projected/0b4d251c-21f8-4b72-99c2-3efe08d34dda-kube-api-access-x4vs8\") pod \"test-model-134dad-kserve-88444b5bf-n8nkf\" (UID: \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\") " pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 04:56:40.965130 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:40.965037    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-model-cache\") pod \"test-model-134dad-kserve-88444b5bf-n8nkf\" (UID: \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\") " pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 04:56:40.965511 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:40.965145    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-tmp-dir\") pod \"test-model-134dad-kserve-88444b5bf-n8nkf\" (UID: \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\") " pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 04:56:40.965511 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:40.965478    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-home\") pod \"test-model-134dad-kserve-88444b5bf-n8nkf\" (UID: \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\") " pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 04:56:40.966983 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:40.966961    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-dshm\") pod \"test-model-134dad-kserve-88444b5bf-n8nkf\" (UID: \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\") " pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 04:56:40.967437 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:40.967415    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b4d251c-21f8-4b72-99c2-3efe08d34dda-tls-certs\") pod \"test-model-134dad-kserve-88444b5bf-n8nkf\" (UID: \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\") " pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 04:56:40.992304 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:40.992283    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4vs8\" (UniqueName: \"kubernetes.io/projected/0b4d251c-21f8-4b72-99c2-3efe08d34dda-kube-api-access-x4vs8\") pod \"test-model-134dad-kserve-88444b5bf-n8nkf\" (UID: \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\") " pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 04:56:41.253126 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:41.212461    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 04:56:42.012303 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:42.012270    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"]
Oct 30 04:56:42.895754 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:42.895719    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf" event={"ID":"0b4d251c-21f8-4b72-99c2-3efe08d34dda","Type":"ContainerStarted","Data":"d172ff117a4bd7e9bb1be2fb2cc43e4a205821195548c31a9646a8f8267bb10b"}
Oct 30 04:56:42.895924 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:42.895766    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf" event={"ID":"0b4d251c-21f8-4b72-99c2-3efe08d34dda","Type":"ContainerStarted","Data":"a15ac09e061203d7db0da2e3eca06d716fe93096d87fdb720213750c2a1395f9"}
Oct 30 04:56:42.938704 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:42.938659    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf" podStartSLOduration=2.938644793 podStartE2EDuration="2.938644793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:42.935103912 +0000 UTC m=+3522.347499486" watchObservedRunningTime="2026-10-30 04:56:42.938644793 +0000 UTC m=+3522.351040366"
Oct 30 04:56:43.215316 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:43.215223    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 04:56:44.565536 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:44.565503    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 04:56:44.578203 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:44.578179    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 04:56:44.922986 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:44.922914    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 04:56:51.214684 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:51.214647    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 04:56:51.226683 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:51.226660    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 04:56:51.970069 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:56:51.970040    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 04:56:55.612388 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:56:55.612350    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:57:14.420453 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:57:14.420368    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:57:17.876608 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:57:17.876572    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp"]
Oct 30 04:57:17.881714 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:57:17.881684    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp"
Oct 30 04:57:17.901275 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:57:17.899893    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp"]
Oct 30 04:57:17.906688 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:57:17.906665    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7700a7-serving-cert\""
Oct 30 04:57:17.914031 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:57:17.913996    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7lq8\" (UniqueName: \"kubernetes.io/projected/318e0d5c-3645-4477-8a63-13849cca77fc-kube-api-access-m7lq8\") pod \"maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp\" (UID: \"318e0d5c-3645-4477-8a63-13849cca77fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp"
Oct 30 04:57:17.914158 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:57:17.914115    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/318e0d5c-3645-4477-8a63-13849cca77fc-maas-api-tls\") pod \"maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp\" (UID: \"318e0d5c-3645-4477-8a63-13849cca77fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp"
Oct 30 04:57:17.914229 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:57:17.914170    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/318e0d5c-3645-4477-8a63-13849cca77fc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp\" (UID: \"318e0d5c-3645-4477-8a63-13849cca77fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp"
Oct 30 04:57:17.914337 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:57:17.914277    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/318e0d5c-3645-4477-8a63-13849cca77fc-metrics-certs\") pod \"maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp\" (UID: \"318e0d5c-3645-4477-8a63-13849cca77fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp"
Oct 30 04:57:18.016262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:57:18.016207    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7lq8\" (UniqueName: \"kubernetes.io/projected/318e0d5c-3645-4477-8a63-13849cca77fc-kube-api-access-m7lq8\") pod \"maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp\" (UID: \"318e0d5c-3645-4477-8a63-13849cca77fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp"
Oct 30 04:57:18.016440 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:57:18.016282    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/318e0d5c-3645-4477-8a63-13849cca77fc-maas-api-tls\") pod \"maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp\" (UID: \"318e0d5c-3645-4477-8a63-13849cca77fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp"
Oct 30 04:57:18.016440 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:57:18.016322    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/318e0d5c-3645-4477-8a63-13849cca77fc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp\" (UID: \"318e0d5c-3645-4477-8a63-13849cca77fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp"
Oct 30 04:57:18.016440 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:57:18.016386    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/318e0d5c-3645-4477-8a63-13849cca77fc-metrics-certs\") pod \"maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp\" (UID: \"318e0d5c-3645-4477-8a63-13849cca77fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp"
Oct 30 04:57:18.016591 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:57:18.016443    4874 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-serving-cert: secret "maas-api-e2e-inf-a-7700a7-serving-cert" not found
Oct 30 04:57:18.016591 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:57:18.016505    4874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/318e0d5c-3645-4477-8a63-13849cca77fc-maas-api-tls podName:318e0d5c-3645-4477-8a63-13849cca77fc nodeName:}" failed. No retries permitted until 2026-10-30 04:57:18.516484186 +0000 UTC m=+3557.928879744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/318e0d5c-3645-4477-8a63-13849cca77fc-maas-api-tls") pod "maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp" (UID: "318e0d5c-3645-4477-8a63-13849cca77fc") : secret "maas-api-e2e-inf-a-7700a7-serving-cert" not found
Oct 30 04:57:18.017264 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:57:18.017208    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/318e0d5c-3645-4477-8a63-13849cca77fc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp\" (UID: \"318e0d5c-3645-4477-8a63-13849cca77fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp"
Oct 30 04:57:18.018954 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:57:18.018929    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/318e0d5c-3645-4477-8a63-13849cca77fc-metrics-certs\") pod \"maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp\" (UID: \"318e0d5c-3645-4477-8a63-13849cca77fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp"
Oct 30 04:57:18.039097 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:57:18.039060    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7lq8\" (UniqueName: \"kubernetes.io/projected/318e0d5c-3645-4477-8a63-13849cca77fc-kube-api-access-m7lq8\") pod \"maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp\" (UID: \"318e0d5c-3645-4477-8a63-13849cca77fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp"
Oct 30 04:57:18.531454 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:57:18.531416    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/318e0d5c-3645-4477-8a63-13849cca77fc-maas-api-tls\") pod \"maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp\" (UID: \"318e0d5c-3645-4477-8a63-13849cca77fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp"
Oct 30 04:57:18.533896 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:57:18.533865    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/318e0d5c-3645-4477-8a63-13849cca77fc-maas-api-tls\") pod \"maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp\" (UID: \"318e0d5c-3645-4477-8a63-13849cca77fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp"
Oct 30 04:57:18.796056 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:57:18.795956    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp"
Oct 30 04:57:19.190199 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:57:19.190166    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp"]
Oct 30 04:57:19.218939 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:57:19.218905    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp" event={"ID":"318e0d5c-3645-4477-8a63-13849cca77fc","Type":"ContainerStarted","Data":"acf164ee0ff1722e03084f03cf4d3c5360300027d0cf6cb331e0379702c0159a"}
Oct 30 04:57:20.231429 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:57:20.231394    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp" event={"ID":"318e0d5c-3645-4477-8a63-13849cca77fc","Type":"ContainerStarted","Data":"6a3a8c5e7c01891c7cc4c1d38dce0f297a229571b5784d705c02afc180b98a5d"}
Oct 30 04:57:20.231987 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:57:20.231961    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp"
Oct 30 04:57:20.277528 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:57:20.277481    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp" podStartSLOduration=2.915787141 podStartE2EDuration="3.277465859s" totalImagesPullingTime="361.678718ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:57:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:57:20.275419425 +0000 UTC m=+3559.687814999" watchObservedRunningTime="2026-10-30 04:57:20.277465859 +0000 UTC m=+3559.689861432"
Oct 30 04:57:25.610613 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:57:25.610572    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:57:26.244580 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:57:26.244547    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp"
Oct 30 04:57:55.610711 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:57:55.610675    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:58:00.881599 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:00.881566    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw"]
Oct 30 04:58:00.886513 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:00.886491    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw"
Oct 30 04:58:00.907661 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:00.907629    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw"]
Oct 30 04:58:00.908877 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:00.908854    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-9f158a-serving-cert\""
Oct 30 04:58:01.045085 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:01.045056    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw\" (UID: \"e51a82fb-9ec5-4fd4-92c7-6ae38029739b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw"
Oct 30 04:58:01.045283 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:01.045102    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-maas-api-tls\") pod \"maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw\" (UID: \"e51a82fb-9ec5-4fd4-92c7-6ae38029739b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw"
Oct 30 04:58:01.045353 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:01.045278    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6pw9\" (UniqueName: \"kubernetes.io/projected/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-kube-api-access-n6pw9\") pod \"maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw\" (UID: \"e51a82fb-9ec5-4fd4-92c7-6ae38029739b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw"
Oct 30 04:58:01.045506 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:01.045491    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-metrics-certs\") pod \"maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw\" (UID: \"e51a82fb-9ec5-4fd4-92c7-6ae38029739b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw"
Oct 30 04:58:01.146941 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:01.146855    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-metrics-certs\") pod \"maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw\" (UID: \"e51a82fb-9ec5-4fd4-92c7-6ae38029739b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw"
Oct 30 04:58:01.147214 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:01.147182    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw\" (UID: \"e51a82fb-9ec5-4fd4-92c7-6ae38029739b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw"
Oct 30 04:58:01.147387 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:01.147268    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-maas-api-tls\") pod \"maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw\" (UID: \"e51a82fb-9ec5-4fd4-92c7-6ae38029739b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw"
Oct 30 04:58:01.147387 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:01.147377    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6pw9\" (UniqueName: \"kubernetes.io/projected/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-kube-api-access-n6pw9\") pod \"maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw\" (UID: \"e51a82fb-9ec5-4fd4-92c7-6ae38029739b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw"
Oct 30 04:58:01.155558 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:01.155528    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw\" (UID: \"e51a82fb-9ec5-4fd4-92c7-6ae38029739b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw"
Oct 30 04:58:01.190266 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:01.176821    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-maas-api-tls\") pod \"maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw\" (UID: \"e51a82fb-9ec5-4fd4-92c7-6ae38029739b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw"
Oct 30 04:58:01.190266 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:01.178666    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-metrics-certs\") pod \"maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw\" (UID: \"e51a82fb-9ec5-4fd4-92c7-6ae38029739b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw"
Oct 30 04:58:01.192090 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:01.191385    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6pw9\" (UniqueName: \"kubernetes.io/projected/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-kube-api-access-n6pw9\") pod \"maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw\" (UID: \"e51a82fb-9ec5-4fd4-92c7-6ae38029739b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw"
Oct 30 04:58:01.203613 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:01.203517    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw"
Oct 30 04:58:01.248620 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:01.248598    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 04:58:01.249813 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:01.249790    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 04:58:01.522969 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:01.522933    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw"]
Oct 30 04:58:01.524337 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:58:01.524308    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode51a82fb_9ec5_4fd4_92c7_6ae38029739b.slice/crio-712a01464cd08e2be4b5cfdeac4680d3fdee04fede550d421aa8384f98770380 WatchSource:0}: Error finding container 712a01464cd08e2be4b5cfdeac4680d3fdee04fede550d421aa8384f98770380: Status 404 returned error can't find the container with id 712a01464cd08e2be4b5cfdeac4680d3fdee04fede550d421aa8384f98770380
Oct 30 04:58:01.630648 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:01.630622    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw" event={"ID":"e51a82fb-9ec5-4fd4-92c7-6ae38029739b","Type":"ContainerStarted","Data":"712a01464cd08e2be4b5cfdeac4680d3fdee04fede550d421aa8384f98770380"}
Oct 30 04:58:02.640615 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:02.640578    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw" event={"ID":"e51a82fb-9ec5-4fd4-92c7-6ae38029739b","Type":"ContainerStarted","Data":"a058f4ec21f99714d09c624336ac3d028768a0341e759b363f8ec9aa2511ff40"}
Oct 30 04:58:02.641011 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:02.640656    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw"
Oct 30 04:58:02.682306 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:02.682253    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw" podStartSLOduration=2.270211428 podStartE2EDuration="2.682225838s" totalImagesPullingTime="412.01441ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:02.680638876 +0000 UTC m=+3602.093034448" watchObservedRunningTime="2026-10-30 04:58:02.682225838 +0000 UTC m=+3602.094621410"
Oct 30 04:58:08.657223 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:08.657190    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw"
Oct 30 04:58:09.126530 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:09.126492    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:58:10.867924 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:10.867892    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"]
Oct 30 04:58:10.868357 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:10.868138    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8" podUID="0d33d6b0-7d09-4361-b56c-da12cddb3e8f" containerName="istio-proxy" containerID="cri-o://7e8b3d2aff6f5b259cf1fc13b4d0045f1de6f40ad6a6645dd3add71b290fa72d" gracePeriod=30
Oct 30 04:58:10.879228 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:10.879198    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"]
Oct 30 04:58:10.879442 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:10.879417    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6" podUID="52058723-4696-41d4-aa59-e3ae0e81e9fd" containerName="istio-proxy" containerID="cri-o://47e063b42e78c3ad0e72efcd8b00d71a5e2264b2a9f2656fe9b1460b5b43ae47" gracePeriod=30
Oct 30 04:58:16.162620 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.162557    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:58:16.198485 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.198461    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:58:16.256257 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.256214    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-kube-api-access-b2fx5\" (UniqueName: \"kubernetes.io/projected/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-kube-api-access-b2fx5\") pod \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") "
Oct 30 04:58:16.256418 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.256293    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-workload-certs\") pod \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") "
Oct 30 04:58:16.256418 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.256319    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-podinfo\") pod \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") "
Oct 30 04:58:16.256552 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.256419    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-workload-socket\") pod \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") "
Oct 30 04:58:16.256617 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.256563    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istiod-ca-cert\") pod \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") "
Oct 30 04:58:16.256617 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.256568    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-workload-certs" pod "0d33d6b0-7d09-4361-b56c-da12cddb3e8f" (UID: "0d33d6b0-7d09-4361-b56c-da12cddb3e8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:58:16.256748 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.256727    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-credential-socket\") pod \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") "
Oct 30 04:58:16.256804 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.256740    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-workload-socket" pod "0d33d6b0-7d09-4361-b56c-da12cddb3e8f" (UID: "0d33d6b0-7d09-4361-b56c-da12cddb3e8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:58:16.256863 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.256810    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-data\") pod \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") "
Oct 30 04:58:16.256922 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.256859    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-envoy\") pod \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") "
Oct 30 04:58:16.256922 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.256890    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-token\" (UniqueName: \"kubernetes.io/projected/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-token\") pod \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\" (UID: \"0d33d6b0-7d09-4361-b56c-da12cddb3e8f\") "
Oct 30 04:58:16.257086 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.257061    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-data" pod "0d33d6b0-7d09-4361-b56c-da12cddb3e8f" (UID: "0d33d6b0-7d09-4361-b56c-da12cddb3e8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:58:16.257487 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.257453    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-credential-socket" pod "0d33d6b0-7d09-4361-b56c-da12cddb3e8f" (UID: "0d33d6b0-7d09-4361-b56c-da12cddb3e8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:58:16.257754 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.257730    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:58:16.257849 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.257761    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:58:16.257849 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.257775    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:58:16.257849 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.257788    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:58:16.258196 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.258168    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istiod-ca-cert" pod "0d33d6b0-7d09-4361-b56c-da12cddb3e8f" (UID: "0d33d6b0-7d09-4361-b56c-da12cddb3e8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:58:16.258758 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.258736    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-kube-api-access-b2fx5" pod "0d33d6b0-7d09-4361-b56c-da12cddb3e8f" (UID: "0d33d6b0-7d09-4361-b56c-da12cddb3e8f"). InnerVolumeSpecName "kube-api-access-b2fx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:58:16.259414 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.259386    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-envoy" pod "0d33d6b0-7d09-4361-b56c-da12cddb3e8f" (UID: "0d33d6b0-7d09-4361-b56c-da12cddb3e8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:58:16.259514 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.259491    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-podinfo" pod "0d33d6b0-7d09-4361-b56c-da12cddb3e8f" (UID: "0d33d6b0-7d09-4361-b56c-da12cddb3e8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:58:16.260138 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.260119    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-token" pod "0d33d6b0-7d09-4361-b56c-da12cddb3e8f" (UID: "0d33d6b0-7d09-4361-b56c-da12cddb3e8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:58:16.358621 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.358589    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52058723-4696-41d4-aa59-e3ae0e81e9fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52058723-4696-41d4-aa59-e3ae0e81e9fd-istiod-ca-cert\") pod \"52058723-4696-41d4-aa59-e3ae0e81e9fd\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") "
Oct 30 04:58:16.358799 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.358632    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-workload-certs\") pod \"52058723-4696-41d4-aa59-e3ae0e81e9fd\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") "
Oct 30 04:58:16.358799 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.358677    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-podinfo\") pod \"52058723-4696-41d4-aa59-e3ae0e81e9fd\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") "
Oct 30 04:58:16.358799 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.358712    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52058723-4696-41d4-aa59-e3ae0e81e9fd-kube-api-access-9sl56\" (UniqueName: \"kubernetes.io/projected/52058723-4696-41d4-aa59-e3ae0e81e9fd-kube-api-access-9sl56\") pod \"52058723-4696-41d4-aa59-e3ae0e81e9fd\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") "
Oct 30 04:58:16.358976 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.358804    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-token\" (UniqueName: \"kubernetes.io/projected/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-token\") pod \"52058723-4696-41d4-aa59-e3ae0e81e9fd\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") "
Oct 30 04:58:16.358976 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.358911    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-workload-socket\") pod \"52058723-4696-41d4-aa59-e3ae0e81e9fd\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") "
Oct 30 04:58:16.358976 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.358943    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-credential-socket\") pod \"52058723-4696-41d4-aa59-e3ae0e81e9fd\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") "
Oct 30 04:58:16.359135 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.359001    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-envoy\") pod \"52058723-4696-41d4-aa59-e3ae0e81e9fd\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") "
Oct 30 04:58:16.359135 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.359032    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-workload-certs" pod "52058723-4696-41d4-aa59-e3ae0e81e9fd" (UID: "52058723-4696-41d4-aa59-e3ae0e81e9fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:58:16.359135 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.359078    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52058723-4696-41d4-aa59-e3ae0e81e9fd-istiod-ca-cert" pod "52058723-4696-41d4-aa59-e3ae0e81e9fd" (UID: "52058723-4696-41d4-aa59-e3ae0e81e9fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:58:16.359135 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.359117    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-data\") pod \"52058723-4696-41d4-aa59-e3ae0e81e9fd\" (UID: \"52058723-4696-41d4-aa59-e3ae0e81e9fd\") "
Oct 30 04:58:16.359371 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.359267    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-workload-socket" pod "52058723-4696-41d4-aa59-e3ae0e81e9fd" (UID: "52058723-4696-41d4-aa59-e3ae0e81e9fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:58:16.359371 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.359341    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-credential-socket" pod "52058723-4696-41d4-aa59-e3ae0e81e9fd" (UID: "52058723-4696-41d4-aa59-e3ae0e81e9fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:58:16.359575 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.359548    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-data" pod "52058723-4696-41d4-aa59-e3ae0e81e9fd" (UID: "52058723-4696-41d4-aa59-e3ae0e81e9fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:58:16.359869 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.359851    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:58:16.359960 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.359873    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:58:16.359960 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.359887    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52058723-4696-41d4-aa59-e3ae0e81e9fd-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:58:16.359960 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.359898    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:58:16.359960 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.359910    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:58:16.359960 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.359922    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:58:16.359960 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.359935    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2fx5\" (UniqueName: \"kubernetes.io/projected/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-kube-api-access-b2fx5\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:58:16.359960 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.359948    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d33d6b0-7d09-4361-b56c-da12cddb3e8f-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:58:16.359960 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.359960    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:58:16.360230 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.359974    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:58:16.361115 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.361083    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52058723-4696-41d4-aa59-e3ae0e81e9fd-kube-api-access-9sl56" pod "52058723-4696-41d4-aa59-e3ae0e81e9fd" (UID: "52058723-4696-41d4-aa59-e3ae0e81e9fd"). InnerVolumeSpecName "kube-api-access-9sl56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:58:16.361183 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.361174    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-envoy" pod "52058723-4696-41d4-aa59-e3ae0e81e9fd" (UID: "52058723-4696-41d4-aa59-e3ae0e81e9fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:58:16.361368 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.361346    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-podinfo" pod "52058723-4696-41d4-aa59-e3ae0e81e9fd" (UID: "52058723-4696-41d4-aa59-e3ae0e81e9fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:58:16.361483 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.361453    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-token" pod "52058723-4696-41d4-aa59-e3ae0e81e9fd" (UID: "52058723-4696-41d4-aa59-e3ae0e81e9fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:58:16.461360 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.461281    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:58:16.461360 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.461311    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:58:16.461360 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.461322    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52058723-4696-41d4-aa59-e3ae0e81e9fd-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:58:16.461360 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.461332    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sl56\" (UniqueName: \"kubernetes.io/projected/52058723-4696-41d4-aa59-e3ae0e81e9fd-kube-api-access-9sl56\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 04:58:16.742192 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.742161    4874 generic.go:356] "Generic (PLEG): container finished" podID="0d33d6b0-7d09-4361-b56c-da12cddb3e8f" containerID="7e8b3d2aff6f5b259cf1fc13b4d0045f1de6f40ad6a6645dd3add71b290fa72d" exitCode=0
Oct 30 04:58:16.742929 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.742348    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"
Oct 30 04:58:16.742929 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.742354    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8" event={"ID":"0d33d6b0-7d09-4361-b56c-da12cddb3e8f","Type":"ContainerDied","Data":"7e8b3d2aff6f5b259cf1fc13b4d0045f1de6f40ad6a6645dd3add71b290fa72d"}
Oct 30 04:58:16.742929 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.742432    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8" event={"ID":"0d33d6b0-7d09-4361-b56c-da12cddb3e8f","Type":"ContainerDied","Data":"65607753b818187b80829c675a0030e88e98ba1aa734dd10c1e667ed8fa3ed58"}
Oct 30 04:58:16.742929 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.742474    4874 scope.go:122] "RemoveContainer" containerID="7e8b3d2aff6f5b259cf1fc13b4d0045f1de6f40ad6a6645dd3add71b290fa72d"
Oct 30 04:58:16.744123 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.744099    4874 generic.go:356] "Generic (PLEG): container finished" podID="52058723-4696-41d4-aa59-e3ae0e81e9fd" containerID="47e063b42e78c3ad0e72efcd8b00d71a5e2264b2a9f2656fe9b1460b5b43ae47" exitCode=0
Oct 30 04:58:16.744266 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.744155    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"
Oct 30 04:58:16.744425 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.744397    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6" event={"ID":"52058723-4696-41d4-aa59-e3ae0e81e9fd","Type":"ContainerDied","Data":"47e063b42e78c3ad0e72efcd8b00d71a5e2264b2a9f2656fe9b1460b5b43ae47"}
Oct 30 04:58:16.744580 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.744452    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6" event={"ID":"52058723-4696-41d4-aa59-e3ae0e81e9fd","Type":"ContainerDied","Data":"a00d6ed1c3292acac316f0a128a991da2f24a8a0872c033be77065f2566396fd"}
Oct 30 04:58:16.753419 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.753398    4874 scope.go:122] "RemoveContainer" containerID="7e8b3d2aff6f5b259cf1fc13b4d0045f1de6f40ad6a6645dd3add71b290fa72d"
Oct 30 04:58:16.753733 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:58:16.753709    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e8b3d2aff6f5b259cf1fc13b4d0045f1de6f40ad6a6645dd3add71b290fa72d\": container with ID starting with 7e8b3d2aff6f5b259cf1fc13b4d0045f1de6f40ad6a6645dd3add71b290fa72d not found: ID does not exist" containerID="7e8b3d2aff6f5b259cf1fc13b4d0045f1de6f40ad6a6645dd3add71b290fa72d"
Oct 30 04:58:16.753845 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.753739    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e8b3d2aff6f5b259cf1fc13b4d0045f1de6f40ad6a6645dd3add71b290fa72d"} err="failed to get container status \"7e8b3d2aff6f5b259cf1fc13b4d0045f1de6f40ad6a6645dd3add71b290fa72d\": rpc error: code = NotFound desc = could not find container \"7e8b3d2aff6f5b259cf1fc13b4d0045f1de6f40ad6a6645dd3add71b290fa72d\": container with ID starting with 7e8b3d2aff6f5b259cf1fc13b4d0045f1de6f40ad6a6645dd3add71b290fa72d not found: ID does not exist"
Oct 30 04:58:16.753845 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.753758    4874 scope.go:122] "RemoveContainer" containerID="47e063b42e78c3ad0e72efcd8b00d71a5e2264b2a9f2656fe9b1460b5b43ae47"
Oct 30 04:58:16.763744 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.763724    4874 scope.go:122] "RemoveContainer" containerID="47e063b42e78c3ad0e72efcd8b00d71a5e2264b2a9f2656fe9b1460b5b43ae47"
Oct 30 04:58:16.764069 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:58:16.764043    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47e063b42e78c3ad0e72efcd8b00d71a5e2264b2a9f2656fe9b1460b5b43ae47\": container with ID starting with 47e063b42e78c3ad0e72efcd8b00d71a5e2264b2a9f2656fe9b1460b5b43ae47 not found: ID does not exist" containerID="47e063b42e78c3ad0e72efcd8b00d71a5e2264b2a9f2656fe9b1460b5b43ae47"
Oct 30 04:58:16.764174 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.764074    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e063b42e78c3ad0e72efcd8b00d71a5e2264b2a9f2656fe9b1460b5b43ae47"} err="failed to get container status \"47e063b42e78c3ad0e72efcd8b00d71a5e2264b2a9f2656fe9b1460b5b43ae47\": rpc error: code = NotFound desc = could not find container \"47e063b42e78c3ad0e72efcd8b00d71a5e2264b2a9f2656fe9b1460b5b43ae47\": container with ID starting with 47e063b42e78c3ad0e72efcd8b00d71a5e2264b2a9f2656fe9b1460b5b43ae47 not found: ID does not exist"
Oct 30 04:58:16.794411 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.794383    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"]
Oct 30 04:58:16.806698 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.806674    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-27xg6"]
Oct 30 04:58:16.830694 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.830671    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"]
Oct 30 04:58:16.834699 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:16.834678    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rr9m8"]
Oct 30 04:58:18.731462 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:18.731430    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d33d6b0-7d09-4361-b56c-da12cddb3e8f" path="/var/lib/kubelet/pods/0d33d6b0-7d09-4361-b56c-da12cddb3e8f/volumes"
Oct 30 04:58:18.732036 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:18.732017    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52058723-4696-41d4-aa59-e3ae0e81e9fd" path="/var/lib/kubelet/pods/52058723-4696-41d4-aa59-e3ae0e81e9fd/volumes"
Oct 30 04:58:19.964329 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:19.964273    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"]
Oct 30 04:58:19.965118 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:19.965097    4874 cpu_manager.go:414] "containerMap: removing container" podUID="52058723-4696-41d4-aa59-e3ae0e81e9fd" containerName="istio-proxy"
Oct 30 04:58:19.965183 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:19.965121    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52058723-4696-41d4-aa59-e3ae0e81e9fd" containerName="istio-proxy"
Oct 30 04:58:19.965183 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:19.965145    4874 cpu_manager.go:414] "containerMap: removing container" podUID="0d33d6b0-7d09-4361-b56c-da12cddb3e8f" containerName="istio-proxy"
Oct 30 04:58:19.965183 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:19.965153    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d33d6b0-7d09-4361-b56c-da12cddb3e8f" containerName="istio-proxy"
Oct 30 04:58:19.965409 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:19.965388    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="52058723-4696-41d4-aa59-e3ae0e81e9fd" containerName="istio-proxy"
Oct 30 04:58:19.965473 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:19.965412    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d33d6b0-7d09-4361-b56c-da12cddb3e8f" containerName="istio-proxy"
Oct 30 04:58:19.969468 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:19.969447    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:19.992798 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:19.992772    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"]
Oct 30 04:58:19.996617 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:19.996595    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e592d7-openshift-default-dockercfg-wfpp6\""
Oct 30 04:58:20.125263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.122540    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzwn5\" (UniqueName: \"kubernetes.io/projected/8ca71311-3f21-47db-98a2-a9f95fbdebe9-kube-api-access-nzwn5\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.125263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.122645    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-envoy\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.125263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.122811    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-token\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.125263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.122902    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-workload-certs\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.125263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.122930    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istiod-ca-cert\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.125263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.123008    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-workload-socket\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.125263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.123044    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-credential-socket\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.125263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.123073    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-data\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.125263 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.123125    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-podinfo\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.315460 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.288187    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-token\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.315782 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.315752    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-workload-certs\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.319054 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.319026    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istiod-ca-cert\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.319357 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.319337    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-workload-socket\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.319491 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.319476    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-credential-socket\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.319607 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.319592    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-data\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.319727 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.319716    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-podinfo\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.319887 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.319875    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzwn5\" (UniqueName: \"kubernetes.io/projected/8ca71311-3f21-47db-98a2-a9f95fbdebe9-kube-api-access-nzwn5\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.320069 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.320005    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-credential-socket\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.320260 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.320010    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istiod-ca-cert\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.320377 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.319759    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-workload-socket\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.320472 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.320023    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-envoy\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.320767 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.320093    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-data\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.320889 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.316767    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-workload-certs\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.354907 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.350671    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-envoy\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.354907 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.351184    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-token\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.366019 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.365925    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzwn5\" (UniqueName: \"kubernetes.io/projected/8ca71311-3f21-47db-98a2-a9f95fbdebe9-kube-api-access-nzwn5\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.375120 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.375056    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-podinfo\") pod \"e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") " pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.584081 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.584008    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:20.909176 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.909149    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"]
Oct 30 04:58:20.910363 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:58:20.910338    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca71311_3f21_47db_98a2_a9f95fbdebe9.slice/crio-fc5a94729b9b3f7166f70b71f783d07a16122ee0860c615f36512deb745ff924 WatchSource:0}: Error finding container fc5a94729b9b3f7166f70b71f783d07a16122ee0860c615f36512deb745ff924: Status 404 returned error can't find the container with id fc5a94729b9b3f7166f70b71f783d07a16122ee0860c615f36512deb745ff924
Oct 30 04:58:20.913148 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.913115    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:58:20.913272 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.913179    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:58:20.913272 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:20.913218    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:58:21.784946 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:21.784913    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6" event={"ID":"8ca71311-3f21-47db-98a2-a9f95fbdebe9","Type":"ContainerStarted","Data":"0c21bad86504624583c3386731842623b64602c0a4b1e7a3d7331545021f642b"}
Oct 30 04:58:21.785361 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:21.784972    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6" event={"ID":"8ca71311-3f21-47db-98a2-a9f95fbdebe9","Type":"ContainerStarted","Data":"fc5a94729b9b3f7166f70b71f783d07a16122ee0860c615f36512deb745ff924"}
Oct 30 04:58:21.842514 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:21.842441    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6" podStartSLOduration=2.842426896 podStartE2EDuration="2.842426896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:21.841863851 +0000 UTC m=+3621.254259416" watchObservedRunningTime="2026-10-30 04:58:21.842426896 +0000 UTC m=+3621.254822471"
Oct 30 04:58:22.584983 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:22.584957    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:22.591286 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:22.591261    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:22.791233 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:22.791199    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:22.792104 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:22.792086    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 04:58:25.609560 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:58:25.609525    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:58:54.001866 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:58:54.001828    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:58:55.613447 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:58:55.613406    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:59:01.774166 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:01.774127    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"]
Oct 30 04:59:01.781207 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:01.781177    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:01.798526 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:01.798496    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"]
Oct 30 04:59:01.831275 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:01.831226    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-26b72f-openshift-default-dockercfg-555w7\""
Oct 30 04:59:01.972344 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:01.971180    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-envoy\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:01.972344 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:01.971369    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4jsg\" (UniqueName: \"kubernetes.io/projected/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-kube-api-access-j4jsg\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:01.972344 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:01.971418    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-data\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:01.972344 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:01.971540    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-token\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:01.972344 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:01.971609    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-podinfo\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:01.972344 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:01.971655    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-workload-socket\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:01.972344 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:01.971711    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-workload-certs\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:01.972344 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:01.971809    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istiod-ca-cert\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:01.972344 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:01.971900    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-credential-socket\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:02.078330 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:02.078293    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-envoy\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:02.080445 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:02.080419    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4jsg\" (UniqueName: \"kubernetes.io/projected/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-kube-api-access-j4jsg\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:02.080925 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:02.080906    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-data\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:02.081469 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:02.081449    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-token\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:02.081877 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:02.081858    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-podinfo\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:02.082456 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:02.082437    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-workload-socket\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:02.099429 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:02.099406    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-workload-certs\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:02.103257 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:02.099977    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istiod-ca-cert\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:02.103571 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:02.103553    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-credential-socket\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:02.113584 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:02.081346    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-data\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:02.113584 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:02.099849    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-workload-certs\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:02.113584 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:02.087637    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-workload-socket\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:02.113584 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:02.104012    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istiod-ca-cert\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:02.113584 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:02.096796    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-podinfo\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:02.113584 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:02.104207    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-credential-socket\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:02.113584 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:02.112003    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-token\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:02.114028 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:02.113722    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-envoy\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:02.117884 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:02.117859    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4jsg\" (UniqueName: \"kubernetes.io/projected/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-kube-api-access-j4jsg\") pod \"e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") " pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:02.396875 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:02.396845    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:02.723841 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:59:02.723805    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b0d7fa_4b81_49a9_9819_2ee76bd5177e.slice/crio-0ef6185afc8e816984dbabcfdd497bce5e2c1ae7a5802a0356a346c4a92a4c31 WatchSource:0}: Error finding container 0ef6185afc8e816984dbabcfdd497bce5e2c1ae7a5802a0356a346c4a92a4c31: Status 404 returned error can't find the container with id 0ef6185afc8e816984dbabcfdd497bce5e2c1ae7a5802a0356a346c4a92a4c31
Oct 30 04:59:02.727167 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:02.727133    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:59:02.727282 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:02.727216    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:59:02.727282 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:02.727268    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:59:02.731197 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:02.731177    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"]
Oct 30 04:59:03.119198 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:03.119160    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj" event={"ID":"54b0d7fa-4b81-49a9-9819-2ee76bd5177e","Type":"ContainerStarted","Data":"08779a31008c91db6218ef983c33d6da7b733e9c2ca22fbba82ba214e54b87f1"}
Oct 30 04:59:03.119669 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:03.119225    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj" event={"ID":"54b0d7fa-4b81-49a9-9819-2ee76bd5177e","Type":"ContainerStarted","Data":"0ef6185afc8e816984dbabcfdd497bce5e2c1ae7a5802a0356a346c4a92a4c31"}
Oct 30 04:59:03.179838 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:03.179779    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj" podStartSLOduration=2.179759953 podStartE2EDuration="2.179759953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:03.178059129 +0000 UTC m=+3662.590454701" watchObservedRunningTime="2026-10-30 04:59:03.179759953 +0000 UTC m=+3662.592155527"
Oct 30 04:59:03.397875 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:03.397799    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:03.403875 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:03.403849    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:04.126514 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:04.126484    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:04.127480 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:04.127460    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 04:59:19.712365 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:19.712328    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb"]
Oct 30 04:59:19.717329 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:19.717309    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb"
Oct 30 04:59:19.735840 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:19.735812    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb"]
Oct 30 04:59:19.737812 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:19.737790    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e592d7-serving-cert\""
Oct 30 04:59:19.823718 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:19.823681    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9735d04-5cad-4502-8f8b-521b47d5c037-maas-api-tls\") pod \"maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb\" (UID: \"a9735d04-5cad-4502-8f8b-521b47d5c037\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb"
Oct 30 04:59:19.824038 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:19.824007    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9735d04-5cad-4502-8f8b-521b47d5c037-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb\" (UID: \"a9735d04-5cad-4502-8f8b-521b47d5c037\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb"
Oct 30 04:59:19.824147 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:19.824133    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9735d04-5cad-4502-8f8b-521b47d5c037-metrics-certs\") pod \"maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb\" (UID: \"a9735d04-5cad-4502-8f8b-521b47d5c037\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb"
Oct 30 04:59:19.824210 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:19.824161    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfk47\" (UniqueName: \"kubernetes.io/projected/a9735d04-5cad-4502-8f8b-521b47d5c037-kube-api-access-sfk47\") pod \"maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb\" (UID: \"a9735d04-5cad-4502-8f8b-521b47d5c037\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb"
Oct 30 04:59:19.925742 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:19.925711    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9735d04-5cad-4502-8f8b-521b47d5c037-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb\" (UID: \"a9735d04-5cad-4502-8f8b-521b47d5c037\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb"
Oct 30 04:59:19.925901 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:19.925756    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9735d04-5cad-4502-8f8b-521b47d5c037-metrics-certs\") pod \"maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb\" (UID: \"a9735d04-5cad-4502-8f8b-521b47d5c037\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb"
Oct 30 04:59:19.925901 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:19.925801    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfk47\" (UniqueName: \"kubernetes.io/projected/a9735d04-5cad-4502-8f8b-521b47d5c037-kube-api-access-sfk47\") pod \"maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb\" (UID: \"a9735d04-5cad-4502-8f8b-521b47d5c037\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb"
Oct 30 04:59:19.926046 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:19.926024    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9735d04-5cad-4502-8f8b-521b47d5c037-maas-api-tls\") pod \"maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb\" (UID: \"a9735d04-5cad-4502-8f8b-521b47d5c037\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb"
Oct 30 04:59:19.926507 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:19.926483    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9735d04-5cad-4502-8f8b-521b47d5c037-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb\" (UID: \"a9735d04-5cad-4502-8f8b-521b47d5c037\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb"
Oct 30 04:59:19.928280 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:19.928232    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9735d04-5cad-4502-8f8b-521b47d5c037-maas-api-tls\") pod \"maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb\" (UID: \"a9735d04-5cad-4502-8f8b-521b47d5c037\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb"
Oct 30 04:59:19.928393 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:19.928291    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9735d04-5cad-4502-8f8b-521b47d5c037-metrics-certs\") pod \"maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb\" (UID: \"a9735d04-5cad-4502-8f8b-521b47d5c037\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb"
Oct 30 04:59:19.949386 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:19.949362    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfk47\" (UniqueName: \"kubernetes.io/projected/a9735d04-5cad-4502-8f8b-521b47d5c037-kube-api-access-sfk47\") pod \"maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb\" (UID: \"a9735d04-5cad-4502-8f8b-521b47d5c037\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb"
Oct 30 04:59:20.037763 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:20.037046    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb"
Oct 30 04:59:20.386293 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:20.386269    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb"]
Oct 30 04:59:20.388364 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:59:20.388335    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9735d04_5cad_4502_8f8b_521b47d5c037.slice/crio-5fd893425b1661475c6f59ad15ef690e1d6f757b12d7d2335e337a9ca62984d8 WatchSource:0}: Error finding container 5fd893425b1661475c6f59ad15ef690e1d6f757b12d7d2335e337a9ca62984d8: Status 404 returned error can't find the container with id 5fd893425b1661475c6f59ad15ef690e1d6f757b12d7d2335e337a9ca62984d8
Oct 30 04:59:21.294843 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:21.294806    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb" event={"ID":"a9735d04-5cad-4502-8f8b-521b47d5c037","Type":"ContainerStarted","Data":"c454f80f25b215c91220f8b1d74ae2ed45c8e54b8dc55a77ac325e42ce38b770"}
Oct 30 04:59:21.295221 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:21.294871    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb"
Oct 30 04:59:21.295221 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:21.294888    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb" event={"ID":"a9735d04-5cad-4502-8f8b-521b47d5c037","Type":"ContainerStarted","Data":"5fd893425b1661475c6f59ad15ef690e1d6f757b12d7d2335e337a9ca62984d8"}
Oct 30 04:59:21.355855 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:21.355795    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb" podStartSLOduration=2.006507665 podStartE2EDuration="2.355774123s" totalImagesPullingTime="349.266458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:21.35226069 +0000 UTC m=+3680.764656282" watchObservedRunningTime="2026-10-30 04:59:21.355774123 +0000 UTC m=+3680.768169697"
Oct 30 04:59:25.611962 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:59:25.611925    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 04:59:27.309471 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:27.309439    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb"
Oct 30 04:59:29.496206 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.496165    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"]
Oct 30 04:59:29.502290 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.502261    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 04:59:29.520253 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.520203    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"]
Oct 30 04:59:29.527505 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.527478    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fd27a9\"/\"default-dockercfg-nrmzf\""
Oct 30 04:59:29.527651 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.527570    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fd27a9\"/\"test-model-fd27a9-kserve-self-signed-certs\""
Oct 30 04:59:29.527651 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.527639    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fd27a9\"/\"openshift-service-ca.crt\""
Oct 30 04:59:29.527800 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.527707    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fd27a9\"/\"kube-root-ca.crt\""
Oct 30 04:59:29.605614 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.605577    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-dshm\") pod \"test-model-fd27a9-kserve-559d8557d9-jqpq7\" (UID: \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\") " pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 04:59:29.605778 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.605615    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-model-cache\") pod \"test-model-fd27a9-kserve-559d8557d9-jqpq7\" (UID: \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\") " pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 04:59:29.605778 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.605704    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26c033bd-b0b9-4b75-ab44-b0ccc7895677-tls-certs\") pod \"test-model-fd27a9-kserve-559d8557d9-jqpq7\" (UID: \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\") " pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 04:59:29.605875 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.605781    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-home\") pod \"test-model-fd27a9-kserve-559d8557d9-jqpq7\" (UID: \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\") " pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 04:59:29.605875 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.605838    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-tmp-dir\") pod \"test-model-fd27a9-kserve-559d8557d9-jqpq7\" (UID: \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\") " pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 04:59:29.605966 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.605945    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg2ns\" (UniqueName: \"kubernetes.io/projected/26c033bd-b0b9-4b75-ab44-b0ccc7895677-kube-api-access-lg2ns\") pod \"test-model-fd27a9-kserve-559d8557d9-jqpq7\" (UID: \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\") " pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 04:59:29.737261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.708152    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-dshm\") pod \"test-model-fd27a9-kserve-559d8557d9-jqpq7\" (UID: \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\") " pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 04:59:29.737261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.708204    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-model-cache\") pod \"test-model-fd27a9-kserve-559d8557d9-jqpq7\" (UID: \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\") " pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 04:59:29.737261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.708273    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26c033bd-b0b9-4b75-ab44-b0ccc7895677-tls-certs\") pod \"test-model-fd27a9-kserve-559d8557d9-jqpq7\" (UID: \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\") " pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 04:59:29.737261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.708346    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-home\") pod \"test-model-fd27a9-kserve-559d8557d9-jqpq7\" (UID: \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\") " pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 04:59:29.737261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.708429    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-tmp-dir\") pod \"test-model-fd27a9-kserve-559d8557d9-jqpq7\" (UID: \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\") " pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 04:59:29.737261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.708513    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg2ns\" (UniqueName: \"kubernetes.io/projected/26c033bd-b0b9-4b75-ab44-b0ccc7895677-kube-api-access-lg2ns\") pod \"test-model-fd27a9-kserve-559d8557d9-jqpq7\" (UID: \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\") " pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 04:59:29.737261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.722898    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-home\") pod \"test-model-fd27a9-kserve-559d8557d9-jqpq7\" (UID: \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\") " pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 04:59:29.737261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.723152    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-tmp-dir\") pod \"test-model-fd27a9-kserve-559d8557d9-jqpq7\" (UID: \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\") " pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 04:59:29.737261 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.723402    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-model-cache\") pod \"test-model-fd27a9-kserve-559d8557d9-jqpq7\" (UID: \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\") " pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 04:59:29.755276 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.753747    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26c033bd-b0b9-4b75-ab44-b0ccc7895677-tls-certs\") pod \"test-model-fd27a9-kserve-559d8557d9-jqpq7\" (UID: \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\") " pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 04:59:29.755276 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.754512    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-dshm\") pod \"test-model-fd27a9-kserve-559d8557d9-jqpq7\" (UID: \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\") " pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 04:59:29.760256 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.756041    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg2ns\" (UniqueName: \"kubernetes.io/projected/26c033bd-b0b9-4b75-ab44-b0ccc7895677-kube-api-access-lg2ns\") pod \"test-model-fd27a9-kserve-559d8557d9-jqpq7\" (UID: \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\") " pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 04:59:29.817718 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:29.817428    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 04:59:30.168995 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:30.168961    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"]
Oct 30 04:59:30.169872 ip-10-0-138-160 kubenswrapper[4874]: W1030 04:59:30.169847    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26c033bd_b0b9_4b75_ab44_b0ccc7895677.slice/crio-ad7ba30699f109e71b1923fba030fe6242e2bdcc0e5b35dfeda553b66c39f32f WatchSource:0}: Error finding container ad7ba30699f109e71b1923fba030fe6242e2bdcc0e5b35dfeda553b66c39f32f: Status 404 returned error can't find the container with id ad7ba30699f109e71b1923fba030fe6242e2bdcc0e5b35dfeda553b66c39f32f
Oct 30 04:59:30.359993 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:30.359953    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7" event={"ID":"26c033bd-b0b9-4b75-ab44-b0ccc7895677","Type":"ContainerStarted","Data":"f55b81f300a3e1e865c5a9d80a4f844352ce6c88d0f4aad0c74f3a38442044c6"}
Oct 30 04:59:30.360165 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:30.360005    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7" event={"ID":"26c033bd-b0b9-4b75-ab44-b0ccc7895677","Type":"ContainerStarted","Data":"ad7ba30699f109e71b1923fba030fe6242e2bdcc0e5b35dfeda553b66c39f32f"}
Oct 30 04:59:30.401443 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:30.401388    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7" podStartSLOduration=1.401375091 podStartE2EDuration="1.401375091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:30.398572553 +0000 UTC m=+3689.810968122" watchObservedRunningTime="2026-10-30 04:59:30.401375091 +0000 UTC m=+3689.813770664"
Oct 30 04:59:30.819270 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:30.819113    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 04:59:39.817921 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:39.817889    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 04:59:39.830694 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:39.830669    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 04:59:40.439432 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:40.439401    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 04:59:51.997262 ip-10-0-138-160 kubenswrapper[4874]: I1030 04:59:51.996962    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 04:59:55.617276 ip-10-0-138-160 kubenswrapper[4874]: E1030 04:59:55.617214    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:00:00.374396 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:00.374361    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk"]
Oct 30 05:00:00.383682 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:00.381194    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk"
Oct 30 05:00:00.413009 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:00.408845    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:00:00.414082 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:00.414048    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk"]
Oct 30 05:00:00.521075 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:00.521041    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcn5x\" (UniqueName: \"kubernetes.io/projected/255f097a-7b21-4089-aec3-b0e3d964317e-kube-api-access-zcn5x\") pod \"maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk\" (UID: \"255f097a-7b21-4089-aec3-b0e3d964317e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk"
Oct 30 05:00:00.521253 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:00.521134    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/255f097a-7b21-4089-aec3-b0e3d964317e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk\" (UID: \"255f097a-7b21-4089-aec3-b0e3d964317e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk"
Oct 30 05:00:00.623300 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:00.623263    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcn5x\" (UniqueName: \"kubernetes.io/projected/255f097a-7b21-4089-aec3-b0e3d964317e-kube-api-access-zcn5x\") pod \"maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk\" (UID: \"255f097a-7b21-4089-aec3-b0e3d964317e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk"
Oct 30 05:00:00.623467 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:00.623330    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/255f097a-7b21-4089-aec3-b0e3d964317e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk\" (UID: \"255f097a-7b21-4089-aec3-b0e3d964317e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk"
Oct 30 05:00:00.624027 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:00.624006    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/255f097a-7b21-4089-aec3-b0e3d964317e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk\" (UID: \"255f097a-7b21-4089-aec3-b0e3d964317e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk"
Oct 30 05:00:00.646647 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:00.646576    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcn5x\" (UniqueName: \"kubernetes.io/projected/255f097a-7b21-4089-aec3-b0e3d964317e-kube-api-access-zcn5x\") pod \"maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk\" (UID: \"255f097a-7b21-4089-aec3-b0e3d964317e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk"
Oct 30 05:00:00.701720 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:00.701693    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk"
Oct 30 05:00:01.022745 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:01.022709    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk"]
Oct 30 05:00:01.028333 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:00:01.028298    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod255f097a_7b21_4089_aec3_b0e3d964317e.slice/crio-8a39645217b2885f2e6be0151ea1f9e5085e8722762649a2439afe82f3660570 WatchSource:0}: Error finding container 8a39645217b2885f2e6be0151ea1f9e5085e8722762649a2439afe82f3660570: Status 404 returned error can't find the container with id 8a39645217b2885f2e6be0151ea1f9e5085e8722762649a2439afe82f3660570
Oct 30 05:00:01.660309 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:01.660272    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk" event={"ID":"255f097a-7b21-4089-aec3-b0e3d964317e","Type":"ContainerStarted","Data":"8a39645217b2885f2e6be0151ea1f9e5085e8722762649a2439afe82f3660570"}
Oct 30 05:00:05.696398 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:05.696368    4874 generic.go:356] "Generic (PLEG): container finished" podID="8ca71311-3f21-47db-98a2-a9f95fbdebe9" containerID="0c21bad86504624583c3386731842623b64602c0a4b1e7a3d7331545021f642b" exitCode=0
Oct 30 05:00:05.696826 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:05.696440    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6" event={"ID":"8ca71311-3f21-47db-98a2-a9f95fbdebe9","Type":"ContainerDied","Data":"0c21bad86504624583c3386731842623b64602c0a4b1e7a3d7331545021f642b"}
Oct 30 05:00:05.696966 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:05.696943    4874 scope.go:122] "RemoveContainer" containerID="0c21bad86504624583c3386731842623b64602c0a4b1e7a3d7331545021f642b"
Oct 30 05:00:05.698736 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:05.698703    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk" event={"ID":"255f097a-7b21-4089-aec3-b0e3d964317e","Type":"ContainerStarted","Data":"d8bc33596cc5f653a0c9c57840d511ecd256dec3d8602332d3fed519463cc926"}
Oct 30 05:00:05.699468 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:05.699433    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 05:00:05.699576 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:05.699528    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 05:00:05.699643 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:05.699576    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 05:00:05.805590 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:05.805081    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk" podStartSLOduration=1.423763723 podStartE2EDuration="5.805063912s" totalImagesPullingTime="4.381300189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:05.804484197 +0000 UTC m=+3725.216879770" watchObservedRunningTime="2026-10-30 05:00:05.805063912 +0000 UTC m=+3725.217459486"
Oct 30 05:00:06.761281 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:06.761219    4874 generic.go:356] "Generic (PLEG): container finished" podID="255f097a-7b21-4089-aec3-b0e3d964317e" containerID="d8bc33596cc5f653a0c9c57840d511ecd256dec3d8602332d3fed519463cc926" exitCode=60
Oct 30 05:00:06.761905 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:06.761887    4874 scope.go:122] "RemoveContainer" containerID="d8bc33596cc5f653a0c9c57840d511ecd256dec3d8602332d3fed519463cc926"
Oct 30 05:00:06.771261 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:06.766616    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk" event={"ID":"255f097a-7b21-4089-aec3-b0e3d964317e","Type":"ContainerDied","Data":"d8bc33596cc5f653a0c9c57840d511ecd256dec3d8602332d3fed519463cc926"}
Oct 30 05:00:06.794734 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:06.794704    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6" event={"ID":"8ca71311-3f21-47db-98a2-a9f95fbdebe9","Type":"ContainerStarted","Data":"5b75178ca380fc44957225360994c9d224faee93c62c15d98c74750c3345911e"}
Oct 30 05:00:07.587313 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:07.585938    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 05:00:07.587313 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:07.586083    4874 patch_prober.go:29] interesting pod/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 05:00:07.587313 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:07.586128    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6" podUID="8ca71311-3f21-47db-98a2-a9f95fbdebe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 05:00:07.791845 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:07.791802    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 05:00:07.803725 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:07.803691    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk" event={"ID":"255f097a-7b21-4089-aec3-b0e3d964317e","Type":"ContainerStarted","Data":"dc3701bd9020be4d55e29f5110b2cbc521dd1ea846bb438e725b3dd676b9e224"}
Oct 30 05:00:08.586266 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:08.585591    4874 patch_prober.go:29] interesting pod/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 05:00:08.586266 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:08.585666    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6" podUID="8ca71311-3f21-47db-98a2-a9f95fbdebe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 05:00:08.812822 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:08.812788    4874 generic.go:356] "Generic (PLEG): container finished" podID="255f097a-7b21-4089-aec3-b0e3d964317e" containerID="dc3701bd9020be4d55e29f5110b2cbc521dd1ea846bb438e725b3dd676b9e224" exitCode=60
Oct 30 05:00:08.813228 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:08.812873    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk" event={"ID":"255f097a-7b21-4089-aec3-b0e3d964317e","Type":"ContainerDied","Data":"dc3701bd9020be4d55e29f5110b2cbc521dd1ea846bb438e725b3dd676b9e224"}
Oct 30 05:00:08.813228 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:08.812932    4874 scope.go:122] "RemoveContainer" containerID="d8bc33596cc5f653a0c9c57840d511ecd256dec3d8602332d3fed519463cc926"
Oct 30 05:00:08.813228 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:08.813218    4874 scope.go:122] "RemoveContainer" containerID="dc3701bd9020be4d55e29f5110b2cbc521dd1ea846bb438e725b3dd676b9e224"
Oct 30 05:00:08.813575 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:00:08.813554    4874 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk_odh-ai-gateway-infra(255f097a-7b21-4089-aec3-b0e3d964317e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk" podUID="255f097a-7b21-4089-aec3-b0e3d964317e"
Oct 30 05:00:09.586269 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:09.585746    4874 patch_prober.go:29] interesting pod/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 05:00:09.586269 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:09.585836    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6" podUID="8ca71311-3f21-47db-98a2-a9f95fbdebe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 05:00:09.821966 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:09.821936    4874 scope.go:122] "RemoveContainer" containerID="dc3701bd9020be4d55e29f5110b2cbc521dd1ea846bb438e725b3dd676b9e224"
Oct 30 05:00:09.822489 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:00:09.822397    4874 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk_odh-ai-gateway-infra(255f097a-7b21-4089-aec3-b0e3d964317e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk" podUID="255f097a-7b21-4089-aec3-b0e3d964317e"
Oct 30 05:00:10.587614 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:10.587577    4874 patch_prober.go:29] interesting pod/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 05:00:10.587892 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:10.587863    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6" podUID="8ca71311-3f21-47db-98a2-a9f95fbdebe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 05:00:11.585260 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:11.584619    4874 patch_prober.go:29] interesting pod/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 05:00:11.585260 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:11.584679    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6" podUID="8ca71311-3f21-47db-98a2-a9f95fbdebe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 05:00:12.588992 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:12.588963    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 05:00:12.590527 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:12.590508    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 05:00:21.761005 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:21.756794    4874 scope.go:122] "RemoveContainer" containerID="dc3701bd9020be4d55e29f5110b2cbc521dd1ea846bb438e725b3dd676b9e224"
Oct 30 05:00:22.954036 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:22.953951    4874 generic.go:356] "Generic (PLEG): container finished" podID="255f097a-7b21-4089-aec3-b0e3d964317e" containerID="a053fc1ecf70afecccb3be51c5222e72983267f0c42a45143d2c31f5942c5eba" exitCode=60
Oct 30 05:00:22.954036 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:22.954024    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk" event={"ID":"255f097a-7b21-4089-aec3-b0e3d964317e","Type":"ContainerDied","Data":"a053fc1ecf70afecccb3be51c5222e72983267f0c42a45143d2c31f5942c5eba"}
Oct 30 05:00:22.954531 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:22.954118    4874 scope.go:122] "RemoveContainer" containerID="dc3701bd9020be4d55e29f5110b2cbc521dd1ea846bb438e725b3dd676b9e224"
Oct 30 05:00:22.954585 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:22.954553    4874 scope.go:122] "RemoveContainer" containerID="a053fc1ecf70afecccb3be51c5222e72983267f0c42a45143d2c31f5942c5eba"
Oct 30 05:00:22.954979 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:00:22.954951    4874 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk_odh-ai-gateway-infra(255f097a-7b21-4089-aec3-b0e3d964317e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk" podUID="255f097a-7b21-4089-aec3-b0e3d964317e"
Oct 30 05:00:24.029983 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:24.029947    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk"]
Oct 30 05:00:24.205170 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:24.205146    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk"
Oct 30 05:00:24.212798 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:24.212776    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/255f097a-7b21-4089-aec3-b0e3d964317e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/255f097a-7b21-4089-aec3-b0e3d964317e-maas-api-service-ca\") pod \"255f097a-7b21-4089-aec3-b0e3d964317e\" (UID: \"255f097a-7b21-4089-aec3-b0e3d964317e\") "
Oct 30 05:00:24.212936 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:24.212869    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/255f097a-7b21-4089-aec3-b0e3d964317e-kube-api-access-zcn5x\" (UniqueName: \"kubernetes.io/projected/255f097a-7b21-4089-aec3-b0e3d964317e-kube-api-access-zcn5x\") pod \"255f097a-7b21-4089-aec3-b0e3d964317e\" (UID: \"255f097a-7b21-4089-aec3-b0e3d964317e\") "
Oct 30 05:00:24.213141 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:24.213117    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/255f097a-7b21-4089-aec3-b0e3d964317e-maas-api-service-ca" pod "255f097a-7b21-4089-aec3-b0e3d964317e" (UID: "255f097a-7b21-4089-aec3-b0e3d964317e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:00:24.213618 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:24.213598    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/255f097a-7b21-4089-aec3-b0e3d964317e-maas-api-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:00:24.214853 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:24.214829    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/255f097a-7b21-4089-aec3-b0e3d964317e-kube-api-access-zcn5x" pod "255f097a-7b21-4089-aec3-b0e3d964317e" (UID: "255f097a-7b21-4089-aec3-b0e3d964317e"). InnerVolumeSpecName "kube-api-access-zcn5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:00:24.315423 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:24.315351    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcn5x\" (UniqueName: \"kubernetes.io/projected/255f097a-7b21-4089-aec3-b0e3d964317e-kube-api-access-zcn5x\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:00:24.859695 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:00:24.859660    4874 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod255f097a_7b21_4089_aec3_b0e3d964317e.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 05:00:25.020779 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:25.020693    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk" event={"ID":"255f097a-7b21-4089-aec3-b0e3d964317e","Type":"ContainerDied","Data":"8a39645217b2885f2e6be0151ea1f9e5085e8722762649a2439afe82f3660570"}
Oct 30 05:00:25.020779 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:25.020729    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk"
Oct 30 05:00:25.020779 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:25.020770    4874 scope.go:122] "RemoveContainer" containerID="a053fc1ecf70afecccb3be51c5222e72983267f0c42a45143d2c31f5942c5eba"
Oct 30 05:00:25.071700 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:25.071673    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk"]
Oct 30 05:00:25.082505 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:25.082481    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e592d7-29888940-8frmk"]
Oct 30 05:00:25.610665 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:00:25.610618    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:00:26.731350 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:26.731311    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="255f097a-7b21-4089-aec3-b0e3d964317e" path="/var/lib/kubelet/pods/255f097a-7b21-4089-aec3-b0e3d964317e/volumes"
Oct 30 05:00:43.182937 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:43.182904    4874 generic.go:356] "Generic (PLEG): container finished" podID="54b0d7fa-4b81-49a9-9819-2ee76bd5177e" containerID="08779a31008c91db6218ef983c33d6da7b733e9c2ca22fbba82ba214e54b87f1" exitCode=0
Oct 30 05:00:43.183506 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:43.182981    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj" event={"ID":"54b0d7fa-4b81-49a9-9819-2ee76bd5177e","Type":"ContainerDied","Data":"08779a31008c91db6218ef983c33d6da7b733e9c2ca22fbba82ba214e54b87f1"}
Oct 30 05:00:43.183586 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:43.183508    4874 scope.go:122] "RemoveContainer" containerID="08779a31008c91db6218ef983c33d6da7b733e9c2ca22fbba82ba214e54b87f1"
Oct 30 05:00:43.185523 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:43.185493    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 05:00:43.185638 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:43.185561    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 05:00:43.185638 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:43.185593    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 05:00:44.197473 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:44.197437    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj" event={"ID":"54b0d7fa-4b81-49a9-9819-2ee76bd5177e","Type":"ContainerStarted","Data":"f5cc58b6414b48fca091651845018b80bfd426adc58ad9872872ed5a0ada161e"}
Oct 30 05:00:44.397939 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:44.397903    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 05:00:44.399107 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:44.399080    4874 patch_prober.go:29] interesting pod/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 05:00:44.399227 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:44.399142    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj" podUID="54b0d7fa-4b81-49a9-9819-2ee76bd5177e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 05:00:45.397868 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:45.397829    4874 patch_prober.go:29] interesting pod/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 05:00:45.398304 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:45.397890    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj" podUID="54b0d7fa-4b81-49a9-9819-2ee76bd5177e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 05:00:46.397552 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:46.397519    4874 patch_prober.go:29] interesting pod/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 05:00:46.397737 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:46.397572    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj" podUID="54b0d7fa-4b81-49a9-9819-2ee76bd5177e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 05:00:47.398090 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:47.398054    4874 patch_prober.go:29] interesting pod/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 05:00:47.398546 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:47.398107    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj" podUID="54b0d7fa-4b81-49a9-9819-2ee76bd5177e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 05:00:48.397475 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:48.397439    4874 patch_prober.go:29] interesting pod/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 05:00:48.397660 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:48.397503    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj" podUID="54b0d7fa-4b81-49a9-9819-2ee76bd5177e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 05:00:49.126539 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:49.126507    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 05:00:49.397834 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:49.397749    4874 patch_prober.go:29] interesting pod/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 05:00:49.397834 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:49.397801    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj" podUID="54b0d7fa-4b81-49a9-9819-2ee76bd5177e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 05:00:50.403013 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:50.402980    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 05:00:50.404094 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:50.404071    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 05:00:55.616802 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:00:55.616754    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:00:56.380313 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:00:56.380274    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:01:08.179617 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:08.179583    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"]
Oct 30 05:01:23.201821 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.201777    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf" podUID="0b4d251c-21f8-4b72-99c2-3efe08d34dda" containerName="main" containerID="cri-o://d172ff117a4bd7e9bb1be2fb2cc43e4a205821195548c31a9646a8f8267bb10b" gracePeriod=45
Oct 30 05:01:23.471882 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.471823    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 05:01:23.545680 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.545652    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-dshm\") pod \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\" (UID: \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\") "
Oct 30 05:01:23.545839 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.545777    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b4d251c-21f8-4b72-99c2-3efe08d34dda-kube-api-access-x4vs8\" (UniqueName: \"kubernetes.io/projected/0b4d251c-21f8-4b72-99c2-3efe08d34dda-kube-api-access-x4vs8\") pod \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\" (UID: \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\") "
Oct 30 05:01:23.545905 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.545840    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b4d251c-21f8-4b72-99c2-3efe08d34dda-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b4d251c-21f8-4b72-99c2-3efe08d34dda-tls-certs\") pod \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\" (UID: \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\") "
Oct 30 05:01:23.545905 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.545866    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-model-cache\") pod \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\" (UID: \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\") "
Oct 30 05:01:23.546024 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.545951    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-home\" (UniqueName: \"kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-home\") pod \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\" (UID: \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\") "
Oct 30 05:01:23.546024 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.545994    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-tmp-dir\") pod \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\" (UID: \"0b4d251c-21f8-4b72-99c2-3efe08d34dda\") "
Oct 30 05:01:23.546327 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.546279    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-model-cache" pod "0b4d251c-21f8-4b72-99c2-3efe08d34dda" (UID: "0b4d251c-21f8-4b72-99c2-3efe08d34dda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:23.546612 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.546582    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-tmp-dir" pod "0b4d251c-21f8-4b72-99c2-3efe08d34dda" (UID: "0b4d251c-21f8-4b72-99c2-3efe08d34dda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:23.546719 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.546648    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-home" pod "0b4d251c-21f8-4b72-99c2-3efe08d34dda" (UID: "0b4d251c-21f8-4b72-99c2-3efe08d34dda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:23.546836 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.546810    4874 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-model-cache\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:23.546836 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.546835    4874 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-home\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:23.546980 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.546847    4874 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-tmp-dir\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:23.548209 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.548176    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b4d251c-21f8-4b72-99c2-3efe08d34dda-tls-certs" pod "0b4d251c-21f8-4b72-99c2-3efe08d34dda" (UID: "0b4d251c-21f8-4b72-99c2-3efe08d34dda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:01:23.548395 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.548369    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b4d251c-21f8-4b72-99c2-3efe08d34dda-kube-api-access-x4vs8" pod "0b4d251c-21f8-4b72-99c2-3efe08d34dda" (UID: "0b4d251c-21f8-4b72-99c2-3efe08d34dda"). InnerVolumeSpecName "kube-api-access-x4vs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:01:23.548478 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.548409    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-dshm" pod "0b4d251c-21f8-4b72-99c2-3efe08d34dda" (UID: "0b4d251c-21f8-4b72-99c2-3efe08d34dda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:23.599320 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.599291    4874 generic.go:356] "Generic (PLEG): container finished" podID="0b4d251c-21f8-4b72-99c2-3efe08d34dda" containerID="d172ff117a4bd7e9bb1be2fb2cc43e4a205821195548c31a9646a8f8267bb10b" exitCode=0
Oct 30 05:01:23.599425 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.599332    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"
Oct 30 05:01:23.599425 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.599370    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf" event={"ID":"0b4d251c-21f8-4b72-99c2-3efe08d34dda","Type":"ContainerDied","Data":"d172ff117a4bd7e9bb1be2fb2cc43e4a205821195548c31a9646a8f8267bb10b"}
Oct 30 05:01:23.599425 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.599415    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf" event={"ID":"0b4d251c-21f8-4b72-99c2-3efe08d34dda","Type":"ContainerDied","Data":"a15ac09e061203d7db0da2e3eca06d716fe93096d87fdb720213750c2a1395f9"}
Oct 30 05:01:23.599540 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.599431    4874 scope.go:122] "RemoveContainer" containerID="d172ff117a4bd7e9bb1be2fb2cc43e4a205821195548c31a9646a8f8267bb10b"
Oct 30 05:01:23.610288 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.610227    4874 scope.go:122] "RemoveContainer" containerID="d172ff117a4bd7e9bb1be2fb2cc43e4a205821195548c31a9646a8f8267bb10b"
Oct 30 05:01:23.610556 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:01:23.610540    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d172ff117a4bd7e9bb1be2fb2cc43e4a205821195548c31a9646a8f8267bb10b\": container with ID starting with d172ff117a4bd7e9bb1be2fb2cc43e4a205821195548c31a9646a8f8267bb10b not found: ID does not exist" containerID="d172ff117a4bd7e9bb1be2fb2cc43e4a205821195548c31a9646a8f8267bb10b"
Oct 30 05:01:23.610613 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.610565    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d172ff117a4bd7e9bb1be2fb2cc43e4a205821195548c31a9646a8f8267bb10b"} err="failed to get container status \"d172ff117a4bd7e9bb1be2fb2cc43e4a205821195548c31a9646a8f8267bb10b\": rpc error: code = NotFound desc = could not find container \"d172ff117a4bd7e9bb1be2fb2cc43e4a205821195548c31a9646a8f8267bb10b\": container with ID starting with d172ff117a4bd7e9bb1be2fb2cc43e4a205821195548c31a9646a8f8267bb10b not found: ID does not exist"
Oct 30 05:01:23.648870 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.648842    4874 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b4d251c-21f8-4b72-99c2-3efe08d34dda-dshm\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:23.648950 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.648874    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4vs8\" (UniqueName: \"kubernetes.io/projected/0b4d251c-21f8-4b72-99c2-3efe08d34dda-kube-api-access-x4vs8\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:23.648950 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.648888    4874 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b4d251c-21f8-4b72-99c2-3efe08d34dda-tls-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:23.650901 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.650880    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"]
Oct 30 05:01:23.655135 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:23.655116    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-134dad/test-model-134dad-kserve-88444b5bf-n8nkf"]
Oct 30 05:01:24.732545 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:24.732511    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b4d251c-21f8-4b72-99c2-3efe08d34dda" path="/var/lib/kubelet/pods/0b4d251c-21f8-4b72-99c2-3efe08d34dda/volumes"
Oct 30 05:01:25.611869 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:01:25.611833    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:01:28.955762 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:28.955724    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"]
Oct 30 05:01:28.956809 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:28.956784    4874 cpu_manager.go:414] "containerMap: removing container" podUID="255f097a-7b21-4089-aec3-b0e3d964317e" containerName="cleanup"
Oct 30 05:01:28.956809 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:28.956809    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="255f097a-7b21-4089-aec3-b0e3d964317e" containerName="cleanup"
Oct 30 05:01:28.956999 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:28.956837    4874 cpu_manager.go:414] "containerMap: removing container" podUID="0b4d251c-21f8-4b72-99c2-3efe08d34dda" containerName="main"
Oct 30 05:01:28.956999 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:28.956845    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b4d251c-21f8-4b72-99c2-3efe08d34dda" containerName="main"
Oct 30 05:01:28.956999 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:28.956920    4874 cpu_manager.go:414] "containerMap: removing container" podUID="255f097a-7b21-4089-aec3-b0e3d964317e" containerName="cleanup"
Oct 30 05:01:28.956999 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:28.956928    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="255f097a-7b21-4089-aec3-b0e3d964317e" containerName="cleanup"
Oct 30 05:01:28.957217 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:28.957084    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="255f097a-7b21-4089-aec3-b0e3d964317e" containerName="cleanup"
Oct 30 05:01:28.957217 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:28.957099    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b4d251c-21f8-4b72-99c2-3efe08d34dda" containerName="main"
Oct 30 05:01:28.957217 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:28.957121    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="255f097a-7b21-4089-aec3-b0e3d964317e" containerName="cleanup"
Oct 30 05:01:28.962891 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:28.962868    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:01:28.991267 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:28.991210    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9f158a\"/\"test-model-9f158a-kserve-self-signed-certs\""
Oct 30 05:01:28.991267 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:28.991211    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9f158a\"/\"openshift-service-ca.crt\""
Oct 30 05:01:28.991438 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:28.991321    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9f158a\"/\"kube-root-ca.crt\""
Oct 30 05:01:28.991438 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:28.991380    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9f158a\"/\"default-dockercfg-v4kws\""
Oct 30 05:01:28.997359 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:28.997330    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"]
Oct 30 05:01:29.026743 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:29.026712    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xft6\" (UniqueName: \"kubernetes.io/projected/761c729c-5767-4b78-b916-0841ad83d9e7-kube-api-access-8xft6\") pod \"test-model-9f158a-kserve-66f8cd44dc-jxz9r\" (UID: \"761c729c-5767-4b78-b916-0841ad83d9e7\") " pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:01:29.026879 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:29.026838    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-model-cache\") pod \"test-model-9f158a-kserve-66f8cd44dc-jxz9r\" (UID: \"761c729c-5767-4b78-b916-0841ad83d9e7\") " pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:01:29.026939 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:29.026896    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-tmp-dir\") pod \"test-model-9f158a-kserve-66f8cd44dc-jxz9r\" (UID: \"761c729c-5767-4b78-b916-0841ad83d9e7\") " pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:01:29.026939 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:29.026922    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/761c729c-5767-4b78-b916-0841ad83d9e7-tls-certs\") pod \"test-model-9f158a-kserve-66f8cd44dc-jxz9r\" (UID: \"761c729c-5767-4b78-b916-0841ad83d9e7\") " pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:01:29.027114 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:29.027095    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-dshm\") pod \"test-model-9f158a-kserve-66f8cd44dc-jxz9r\" (UID: \"761c729c-5767-4b78-b916-0841ad83d9e7\") " pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:01:29.027212 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:29.027198    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-home\") pod \"test-model-9f158a-kserve-66f8cd44dc-jxz9r\" (UID: \"761c729c-5767-4b78-b916-0841ad83d9e7\") " pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:01:29.152664 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:29.152624    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xft6\" (UniqueName: \"kubernetes.io/projected/761c729c-5767-4b78-b916-0841ad83d9e7-kube-api-access-8xft6\") pod \"test-model-9f158a-kserve-66f8cd44dc-jxz9r\" (UID: \"761c729c-5767-4b78-b916-0841ad83d9e7\") " pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:01:29.153039 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:29.153018    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-model-cache\") pod \"test-model-9f158a-kserve-66f8cd44dc-jxz9r\" (UID: \"761c729c-5767-4b78-b916-0841ad83d9e7\") " pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:01:29.170902 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:29.170870    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-tmp-dir\") pod \"test-model-9f158a-kserve-66f8cd44dc-jxz9r\" (UID: \"761c729c-5767-4b78-b916-0841ad83d9e7\") " pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:01:29.171640 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:29.171617    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/761c729c-5767-4b78-b916-0841ad83d9e7-tls-certs\") pod \"test-model-9f158a-kserve-66f8cd44dc-jxz9r\" (UID: \"761c729c-5767-4b78-b916-0841ad83d9e7\") " pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:01:29.172025 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:29.171976    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-dshm\") pod \"test-model-9f158a-kserve-66f8cd44dc-jxz9r\" (UID: \"761c729c-5767-4b78-b916-0841ad83d9e7\") " pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:01:29.176820 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:29.176778    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-home\") pod \"test-model-9f158a-kserve-66f8cd44dc-jxz9r\" (UID: \"761c729c-5767-4b78-b916-0841ad83d9e7\") " pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:01:29.210559 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:29.171535    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-tmp-dir\") pod \"test-model-9f158a-kserve-66f8cd44dc-jxz9r\" (UID: \"761c729c-5767-4b78-b916-0841ad83d9e7\") " pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:01:29.212178 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:29.170737    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-model-cache\") pod \"test-model-9f158a-kserve-66f8cd44dc-jxz9r\" (UID: \"761c729c-5767-4b78-b916-0841ad83d9e7\") " pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:01:29.212378 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:29.177579    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-home\") pod \"test-model-9f158a-kserve-66f8cd44dc-jxz9r\" (UID: \"761c729c-5767-4b78-b916-0841ad83d9e7\") " pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:01:29.222011 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:29.221979    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xft6\" (UniqueName: \"kubernetes.io/projected/761c729c-5767-4b78-b916-0841ad83d9e7-kube-api-access-8xft6\") pod \"test-model-9f158a-kserve-66f8cd44dc-jxz9r\" (UID: \"761c729c-5767-4b78-b916-0841ad83d9e7\") " pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:01:29.225968 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:29.225942    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/761c729c-5767-4b78-b916-0841ad83d9e7-tls-certs\") pod \"test-model-9f158a-kserve-66f8cd44dc-jxz9r\" (UID: \"761c729c-5767-4b78-b916-0841ad83d9e7\") " pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:01:29.232277 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:29.226403    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-dshm\") pod \"test-model-9f158a-kserve-66f8cd44dc-jxz9r\" (UID: \"761c729c-5767-4b78-b916-0841ad83d9e7\") " pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:01:29.279859 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:29.279569    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:01:29.615774 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:29.615744    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"]
Oct 30 05:01:29.616317 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:01:29.616291    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod761c729c_5767_4b78_b916_0841ad83d9e7.slice/crio-f26a680e106c062331e1bc294cf31d1de25245898f89e5ebdc6bb5bb5a319557 WatchSource:0}: Error finding container f26a680e106c062331e1bc294cf31d1de25245898f89e5ebdc6bb5bb5a319557: Status 404 returned error can't find the container with id f26a680e106c062331e1bc294cf31d1de25245898f89e5ebdc6bb5bb5a319557
Oct 30 05:01:29.648289 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:29.648263    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r" event={"ID":"761c729c-5767-4b78-b916-0841ad83d9e7","Type":"ContainerStarted","Data":"f26a680e106c062331e1bc294cf31d1de25245898f89e5ebdc6bb5bb5a319557"}
Oct 30 05:01:30.656739 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:30.656700    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r" event={"ID":"761c729c-5767-4b78-b916-0841ad83d9e7","Type":"ContainerStarted","Data":"783770281ff2b7ae351180a19f36274df88ba4751f36c43b72c02b6bfdcd7eaf"}
Oct 30 05:01:30.698985 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:30.698937    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r" podStartSLOduration=2.698923855 podStartE2EDuration="2.698923855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:01:30.696502566 +0000 UTC m=+3810.108898139" watchObservedRunningTime="2026-10-30 05:01:30.698923855 +0000 UTC m=+3810.111319427"
Oct 30 05:01:31.280838 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:31.280801    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:01:34.375990 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:34.375953    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"]
Oct 30 05:01:39.280399 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:39.280354    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:01:39.293031 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:39.293004    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:01:39.581312 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:39.570774    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"]
Oct 30 05:01:39.581312 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:39.571058    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss" podUID="6ac7a2fc-ad35-41f6-babc-297ab9f7a97e" containerName="istio-proxy" containerID="cri-o://b4dd3c7c14e3397330f15c411b606d08d74b7110247abfd86d391d229d807f4d" gracePeriod=30
Oct 30 05:01:39.780266 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:39.780221    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:01:43.341777 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:43.341687    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"]
Oct 30 05:01:43.342180 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:43.341908    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn" podUID="dd40b6ca-451d-4750-a621-920fa4d7425d" containerName="istio-proxy" containerID="cri-o://140e2cce000819ab5f85f856edd6f82b57c86dba05951825d982caab1bb2b8d5" gracePeriod=30
Oct 30 05:01:43.486053 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:43.486019    4874 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:01:43.486215 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:43.486085    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss" podUID="6ac7a2fc-ad35-41f6-babc-297ab9f7a97e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:01:44.831930 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.831901    4874 generic.go:356] "Generic (PLEG): container finished" podID="6ac7a2fc-ad35-41f6-babc-297ab9f7a97e" containerID="b4dd3c7c14e3397330f15c411b606d08d74b7110247abfd86d391d229d807f4d" exitCode=0
Oct 30 05:01:44.832312 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.831974    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss" event={"ID":"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e","Type":"ContainerDied","Data":"b4dd3c7c14e3397330f15c411b606d08d74b7110247abfd86d391d229d807f4d"}
Oct 30 05:01:44.832312 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.832029    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss" event={"ID":"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e","Type":"ContainerDied","Data":"a9a41c74e6c75a1b4cb83230bd3f09e5cd0eba4c6ca507ca8c89608affabb287"}
Oct 30 05:01:44.832312 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.832048    4874 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9a41c74e6c75a1b4cb83230bd3f09e5cd0eba4c6ca507ca8c89608affabb287"
Oct 30 05:01:44.832312 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.832061    4874 scope.go:122] "RemoveContainer" containerID="36e5bfe5090d29adf31c44d42828df62d4b0565b72a3a848897d9f13298fb244"
Oct 30 05:01:44.847496 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:01:44.847471    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36e5bfe5090d29adf31c44d42828df62d4b0565b72a3a848897d9f13298fb244\": container with ID starting with 36e5bfe5090d29adf31c44d42828df62d4b0565b72a3a848897d9f13298fb244 not found: ID does not exist" containerID="36e5bfe5090d29adf31c44d42828df62d4b0565b72a3a848897d9f13298fb244"
Oct 30 05:01:44.851033 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.851015    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 05:01:44.952739 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.952706    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-workload-socket\") pod \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") "
Oct 30 05:01:44.952906 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.952815    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-credential-socket\") pod \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") "
Oct 30 05:01:44.952906 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.952865    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-data\") pod \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") "
Oct 30 05:01:44.953019 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.952906    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-kube-api-access-xnqct\" (UniqueName: \"kubernetes.io/projected/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-kube-api-access-xnqct\") pod \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") "
Oct 30 05:01:44.953078 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.953034    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-workload-socket" pod "6ac7a2fc-ad35-41f6-babc-297ab9f7a97e" (UID: "6ac7a2fc-ad35-41f6-babc-297ab9f7a97e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:44.953078 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.953053    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-workload-certs\") pod \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") "
Oct 30 05:01:44.953191 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.953089    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-podinfo\") pod \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") "
Oct 30 05:01:44.953191 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.953156    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-token\" (UniqueName: \"kubernetes.io/projected/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-token\") pod \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") "
Oct 30 05:01:44.953191 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.953161    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-credential-socket" pod "6ac7a2fc-ad35-41f6-babc-297ab9f7a97e" (UID: "6ac7a2fc-ad35-41f6-babc-297ab9f7a97e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:44.953413 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.953200    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-envoy\") pod \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") "
Oct 30 05:01:44.953413 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.953279    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istiod-ca-cert\") pod \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\" (UID: \"6ac7a2fc-ad35-41f6-babc-297ab9f7a97e\") "
Oct 30 05:01:44.953920 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.953819    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-data" pod "6ac7a2fc-ad35-41f6-babc-297ab9f7a97e" (UID: "6ac7a2fc-ad35-41f6-babc-297ab9f7a97e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:44.953920 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.953874    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-workload-certs" pod "6ac7a2fc-ad35-41f6-babc-297ab9f7a97e" (UID: "6ac7a2fc-ad35-41f6-babc-297ab9f7a97e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:44.953920 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.953894    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istiod-ca-cert" pod "6ac7a2fc-ad35-41f6-babc-297ab9f7a97e" (UID: "6ac7a2fc-ad35-41f6-babc-297ab9f7a97e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:01:44.954088 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.954054    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:44.954088 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.954074    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:44.954192 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.954088    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:44.954192 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.954100    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:44.954192 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.954113    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:44.955939 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.955912    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-podinfo" pod "6ac7a2fc-ad35-41f6-babc-297ab9f7a97e" (UID: "6ac7a2fc-ad35-41f6-babc-297ab9f7a97e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:01:44.957273 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.956532    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-kube-api-access-xnqct" pod "6ac7a2fc-ad35-41f6-babc-297ab9f7a97e" (UID: "6ac7a2fc-ad35-41f6-babc-297ab9f7a97e"). InnerVolumeSpecName "kube-api-access-xnqct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:01:44.957273 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.956643    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-token" pod "6ac7a2fc-ad35-41f6-babc-297ab9f7a97e" (UID: "6ac7a2fc-ad35-41f6-babc-297ab9f7a97e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:01:44.957823 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:44.957792    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-envoy" pod "6ac7a2fc-ad35-41f6-babc-297ab9f7a97e" (UID: "6ac7a2fc-ad35-41f6-babc-297ab9f7a97e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:45.108410 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:45.108305    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:45.108754 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:45.108736    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:45.108888 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:45.108876    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:45.109006 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:45.108993    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnqct\" (UniqueName: \"kubernetes.io/projected/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e-kube-api-access-xnqct\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:45.840148 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:45.840118    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"
Oct 30 05:01:45.903260 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:45.903220    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"]
Oct 30 05:01:45.915459 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:45.915435    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-wv7ss"]
Oct 30 05:01:46.731203 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:46.731171    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ac7a2fc-ad35-41f6-babc-297ab9f7a97e" path="/var/lib/kubelet/pods/6ac7a2fc-ad35-41f6-babc-297ab9f7a97e/volumes"
Oct 30 05:01:48.071261 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.071220    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:01:48.071644 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.071295    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn" podUID="dd40b6ca-451d-4750-a621-920fa4d7425d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:01:48.630056 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.630025    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 05:01:48.693499 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.693469    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-workload-socket\") pod \"dd40b6ca-451d-4750-a621-920fa4d7425d\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") "
Oct 30 05:01:48.693684 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.693513    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-workload-certs\") pod \"dd40b6ca-451d-4750-a621-920fa4d7425d\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") "
Oct 30 05:01:48.693684 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.693544    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-podinfo\") pod \"dd40b6ca-451d-4750-a621-920fa4d7425d\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") "
Oct 30 05:01:48.693684 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.693657    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-data\") pod \"dd40b6ca-451d-4750-a621-920fa4d7425d\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") "
Oct 30 05:01:48.693878 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.693754    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd40b6ca-451d-4750-a621-920fa4d7425d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd40b6ca-451d-4750-a621-920fa4d7425d-istiod-ca-cert\") pod \"dd40b6ca-451d-4750-a621-920fa4d7425d\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") "
Oct 30 05:01:48.693988 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.693959    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-workload-socket" pod "dd40b6ca-451d-4750-a621-920fa4d7425d" (UID: "dd40b6ca-451d-4750-a621-920fa4d7425d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:48.694090 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.694050    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-workload-certs" pod "dd40b6ca-451d-4750-a621-920fa4d7425d" (UID: "dd40b6ca-451d-4750-a621-920fa4d7425d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:48.694154 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.694137    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-data" pod "dd40b6ca-451d-4750-a621-920fa4d7425d" (UID: "dd40b6ca-451d-4750-a621-920fa4d7425d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:48.694262 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.694228    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd40b6ca-451d-4750-a621-920fa4d7425d-kube-api-access-dwrpb\" (UniqueName: \"kubernetes.io/projected/dd40b6ca-451d-4750-a621-920fa4d7425d-kube-api-access-dwrpb\") pod \"dd40b6ca-451d-4750-a621-920fa4d7425d\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") "
Oct 30 05:01:48.694399 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.694348    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd40b6ca-451d-4750-a621-920fa4d7425d-istiod-ca-cert" pod "dd40b6ca-451d-4750-a621-920fa4d7425d" (UID: "dd40b6ca-451d-4750-a621-920fa4d7425d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:01:48.694399 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.694374    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-credential-socket\") pod \"dd40b6ca-451d-4750-a621-920fa4d7425d\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") "
Oct 30 05:01:48.694536 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.694514    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-envoy\") pod \"dd40b6ca-451d-4750-a621-920fa4d7425d\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") "
Oct 30 05:01:48.694707 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.694546    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-token\" (UniqueName: \"kubernetes.io/projected/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-token\") pod \"dd40b6ca-451d-4750-a621-920fa4d7425d\" (UID: \"dd40b6ca-451d-4750-a621-920fa4d7425d\") "
Oct 30 05:01:48.694707 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.694615    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-credential-socket" pod "dd40b6ca-451d-4750-a621-920fa4d7425d" (UID: "dd40b6ca-451d-4750-a621-920fa4d7425d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:48.695182 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.695159    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:48.695303 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.695186    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:48.695303 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.695201    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:48.695303 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.695214    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd40b6ca-451d-4750-a621-920fa4d7425d-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:48.695303 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.695228    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:48.696308 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.696281    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-podinfo" pod "dd40b6ca-451d-4750-a621-920fa4d7425d" (UID: "dd40b6ca-451d-4750-a621-920fa4d7425d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:01:48.696703 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.696681    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd40b6ca-451d-4750-a621-920fa4d7425d-kube-api-access-dwrpb" pod "dd40b6ca-451d-4750-a621-920fa4d7425d" (UID: "dd40b6ca-451d-4750-a621-920fa4d7425d"). InnerVolumeSpecName "kube-api-access-dwrpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:01:48.696787 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.696766    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-token" pod "dd40b6ca-451d-4750-a621-920fa4d7425d" (UID: "dd40b6ca-451d-4750-a621-920fa4d7425d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:01:48.697063 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.697042    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-envoy" pod "dd40b6ca-451d-4750-a621-920fa4d7425d" (UID: "dd40b6ca-451d-4750-a621-920fa4d7425d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:48.803928 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.803893    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:48.804374 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.804322    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:48.804756 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.804729    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd40b6ca-451d-4750-a621-920fa4d7425d-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:48.804756 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.804755    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwrpb\" (UniqueName: \"kubernetes.io/projected/dd40b6ca-451d-4750-a621-920fa4d7425d-kube-api-access-dwrpb\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:48.890445 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.890368    4874 generic.go:356] "Generic (PLEG): container finished" podID="dd40b6ca-451d-4750-a621-920fa4d7425d" containerID="140e2cce000819ab5f85f856edd6f82b57c86dba05951825d982caab1bb2b8d5" exitCode=0
Oct 30 05:01:48.890445 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.890408    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn" event={"ID":"dd40b6ca-451d-4750-a621-920fa4d7425d","Type":"ContainerDied","Data":"140e2cce000819ab5f85f856edd6f82b57c86dba05951825d982caab1bb2b8d5"}
Oct 30 05:01:48.890445 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.890416    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"
Oct 30 05:01:48.890696 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.890453    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn" event={"ID":"dd40b6ca-451d-4750-a621-920fa4d7425d","Type":"ContainerDied","Data":"78b86bfc69f18869406e7414849d9bd9b531401f5e4cc33224b8d02bf6e98a9d"}
Oct 30 05:01:48.890696 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.890474    4874 scope.go:122] "RemoveContainer" containerID="140e2cce000819ab5f85f856edd6f82b57c86dba05951825d982caab1bb2b8d5"
Oct 30 05:01:48.900810 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.900790    4874 scope.go:122] "RemoveContainer" containerID="140e2cce000819ab5f85f856edd6f82b57c86dba05951825d982caab1bb2b8d5"
Oct 30 05:01:48.901048 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:01:48.901026    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"140e2cce000819ab5f85f856edd6f82b57c86dba05951825d982caab1bb2b8d5\": container with ID starting with 140e2cce000819ab5f85f856edd6f82b57c86dba05951825d982caab1bb2b8d5 not found: ID does not exist" containerID="140e2cce000819ab5f85f856edd6f82b57c86dba05951825d982caab1bb2b8d5"
Oct 30 05:01:48.901118 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.901059    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"140e2cce000819ab5f85f856edd6f82b57c86dba05951825d982caab1bb2b8d5"} err="failed to get container status \"140e2cce000819ab5f85f856edd6f82b57c86dba05951825d982caab1bb2b8d5\": rpc error: code = NotFound desc = could not find container \"140e2cce000819ab5f85f856edd6f82b57c86dba05951825d982caab1bb2b8d5\": container with ID starting with 140e2cce000819ab5f85f856edd6f82b57c86dba05951825d982caab1bb2b8d5 not found: ID does not exist"
Oct 30 05:01:48.941415 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.941390    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"]
Oct 30 05:01:48.945789 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:48.945765    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-m4jdn"]
Oct 30 05:01:49.397105 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.397070    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr" podUID="f1e88b87-85fa-4cce-ad5d-927bcddc23e9" containerName="main" containerID="cri-o://eda64d8ac7985380372c5fade0a90c6ba99fbaf13ea22c80073405de2e21e481" gracePeriod=45
Oct 30 05:01:49.717651 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.717615    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 05:01:49.773847 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.773817    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"]
Oct 30 05:01:49.774030 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.773989    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4" podUID="90a79cd0-bb0f-40a4-bb6d-5099288a42e8" containerName="istio-proxy" containerID="cri-o://78b2a4b3b1484fad04de8b1d0911a6b074331c3049cdf98f47ce918f93cd49ba" gracePeriod=30
Oct 30 05:01:49.816933 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.816900    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-tls-certs\") pod \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\" (UID: \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\") "
Oct 30 05:01:49.817109 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.816962    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-home\" (UniqueName: \"kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-home\") pod \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\" (UID: \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\") "
Oct 30 05:01:49.817109 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.817050    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-model-cache\") pod \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\" (UID: \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\") "
Oct 30 05:01:49.817214 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.817196    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-home" pod "f1e88b87-85fa-4cce-ad5d-927bcddc23e9" (UID: "f1e88b87-85fa-4cce-ad5d-927bcddc23e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:49.817311 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.817294    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-tmp-dir\") pod \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\" (UID: \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\") "
Oct 30 05:01:49.817370 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.817318    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-model-cache" pod "f1e88b87-85fa-4cce-ad5d-927bcddc23e9" (UID: "f1e88b87-85fa-4cce-ad5d-927bcddc23e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:49.817421 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.817379    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-dshm\") pod \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\" (UID: \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\") "
Oct 30 05:01:49.817482 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.817425    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-kube-api-access-ftw6c\" (UniqueName: \"kubernetes.io/projected/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-kube-api-access-ftw6c\") pod \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\" (UID: \"f1e88b87-85fa-4cce-ad5d-927bcddc23e9\") "
Oct 30 05:01:49.817536 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.817489    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-tmp-dir" pod "f1e88b87-85fa-4cce-ad5d-927bcddc23e9" (UID: "f1e88b87-85fa-4cce-ad5d-927bcddc23e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:49.818529 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.818314    4874 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-tmp-dir\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:49.818529 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.818390    4874 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-home\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:49.818529 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.818412    4874 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-model-cache\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:49.819751 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.819724    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-dshm" pod "f1e88b87-85fa-4cce-ad5d-927bcddc23e9" (UID: "f1e88b87-85fa-4cce-ad5d-927bcddc23e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:49.819864 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.819789    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-tls-certs" pod "f1e88b87-85fa-4cce-ad5d-927bcddc23e9" (UID: "f1e88b87-85fa-4cce-ad5d-927bcddc23e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:01:49.820020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.820003    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-kube-api-access-ftw6c" pod "f1e88b87-85fa-4cce-ad5d-927bcddc23e9" (UID: "f1e88b87-85fa-4cce-ad5d-927bcddc23e9"). InnerVolumeSpecName "kube-api-access-ftw6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:01:49.900042 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.900010    4874 generic.go:356] "Generic (PLEG): container finished" podID="f1e88b87-85fa-4cce-ad5d-927bcddc23e9" containerID="eda64d8ac7985380372c5fade0a90c6ba99fbaf13ea22c80073405de2e21e481" exitCode=0
Oct 30 05:01:49.900277 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.900055    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"
Oct 30 05:01:49.900277 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.900097    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr" event={"ID":"f1e88b87-85fa-4cce-ad5d-927bcddc23e9","Type":"ContainerDied","Data":"eda64d8ac7985380372c5fade0a90c6ba99fbaf13ea22c80073405de2e21e481"}
Oct 30 05:01:49.900277 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.900146    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr" event={"ID":"f1e88b87-85fa-4cce-ad5d-927bcddc23e9","Type":"ContainerDied","Data":"f00c9e4ff0f867b5f0c1bf2c93a9e592cb41d1f1dbcf7269c7077db8632d4c26"}
Oct 30 05:01:49.900277 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.900170    4874 scope.go:122] "RemoveContainer" containerID="eda64d8ac7985380372c5fade0a90c6ba99fbaf13ea22c80073405de2e21e481"
Oct 30 05:01:49.911188 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.911169    4874 scope.go:122] "RemoveContainer" containerID="eda64d8ac7985380372c5fade0a90c6ba99fbaf13ea22c80073405de2e21e481"
Oct 30 05:01:49.911457 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:01:49.911437    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eda64d8ac7985380372c5fade0a90c6ba99fbaf13ea22c80073405de2e21e481\": container with ID starting with eda64d8ac7985380372c5fade0a90c6ba99fbaf13ea22c80073405de2e21e481 not found: ID does not exist" containerID="eda64d8ac7985380372c5fade0a90c6ba99fbaf13ea22c80073405de2e21e481"
Oct 30 05:01:49.911502 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.911464    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda64d8ac7985380372c5fade0a90c6ba99fbaf13ea22c80073405de2e21e481"} err="failed to get container status \"eda64d8ac7985380372c5fade0a90c6ba99fbaf13ea22c80073405de2e21e481\": rpc error: code = NotFound desc = could not find container \"eda64d8ac7985380372c5fade0a90c6ba99fbaf13ea22c80073405de2e21e481\": container with ID starting with eda64d8ac7985380372c5fade0a90c6ba99fbaf13ea22c80073405de2e21e481 not found: ID does not exist"
Oct 30 05:01:49.919859 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.919838    4874 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-dshm\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:49.919859 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.919860    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftw6c\" (UniqueName: \"kubernetes.io/projected/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-kube-api-access-ftw6c\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:49.919978 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.919870    4874 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e88b87-85fa-4cce-ad5d-927bcddc23e9-tls-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:49.949930 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.949878    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"]
Oct 30 05:01:49.954225 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:49.954205    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f9cd17/test-model-f9cd17-kserve-865d955fc9-mbgkr"]
Oct 30 05:01:50.730166 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:50.730129    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd40b6ca-451d-4750-a621-920fa4d7425d" path="/var/lib/kubelet/pods/dd40b6ca-451d-4750-a621-920fa4d7425d/volumes"
Oct 30 05:01:50.730609 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:50.730591    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1e88b87-85fa-4cce-ad5d-927bcddc23e9" path="/var/lib/kubelet/pods/f1e88b87-85fa-4cce-ad5d-927bcddc23e9/volumes"
Oct 30 05:01:54.936991 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:54.936959    4874 generic.go:356] "Generic (PLEG): container finished" podID="90a79cd0-bb0f-40a4-bb6d-5099288a42e8" containerID="78b2a4b3b1484fad04de8b1d0911a6b074331c3049cdf98f47ce918f93cd49ba" exitCode=0
Oct 30 05:01:54.937527 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:54.937018    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4" event={"ID":"90a79cd0-bb0f-40a4-bb6d-5099288a42e8","Type":"ContainerDied","Data":"78b2a4b3b1484fad04de8b1d0911a6b074331c3049cdf98f47ce918f93cd49ba"}
Oct 30 05:01:54.937527 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:54.937067    4874 scope.go:122] "RemoveContainer" containerID="07ee46e07e2c4e12d1eb627dd0bb43c0673052e9861651dc00899a273a6e5e07"
Oct 30 05:01:55.053962 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.053927    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 05:01:55.204383 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.204302    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-workload-socket\") pod \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") "
Oct 30 05:01:55.204383 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.204364    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-envoy\") pod \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") "
Oct 30 05:01:55.204591 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.204479    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-token\" (UniqueName: \"kubernetes.io/projected/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-token\") pod \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") "
Oct 30 05:01:55.204591 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.204573    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-workload-socket" pod "90a79cd0-bb0f-40a4-bb6d-5099288a42e8" (UID: "90a79cd0-bb0f-40a4-bb6d-5099288a42e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:55.204717 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.204601    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-credential-socket\") pod \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") "
Oct 30 05:01:55.204717 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.204644    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-workload-certs\") pod \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") "
Oct 30 05:01:55.204717 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.204697    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-data\") pod \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") "
Oct 30 05:01:55.204864 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.204820    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-podinfo\") pod \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") "
Oct 30 05:01:55.204864 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.204853    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-credential-socket" pod "90a79cd0-bb0f-40a4-bb6d-5099288a42e8" (UID: "90a79cd0-bb0f-40a4-bb6d-5099288a42e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:55.204985 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.204856    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istiod-ca-cert\") pod \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") "
Oct 30 05:01:55.204985 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.204880    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-workload-certs" pod "90a79cd0-bb0f-40a4-bb6d-5099288a42e8" (UID: "90a79cd0-bb0f-40a4-bb6d-5099288a42e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:55.204985 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.204941    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-kube-api-access-9mlhl\" (UniqueName: \"kubernetes.io/projected/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-kube-api-access-9mlhl\") pod \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\" (UID: \"90a79cd0-bb0f-40a4-bb6d-5099288a42e8\") "
Oct 30 05:01:55.205152 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.205090    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-data" pod "90a79cd0-bb0f-40a4-bb6d-5099288a42e8" (UID: "90a79cd0-bb0f-40a4-bb6d-5099288a42e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:55.205212 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.205194    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istiod-ca-cert" pod "90a79cd0-bb0f-40a4-bb6d-5099288a42e8" (UID: "90a79cd0-bb0f-40a4-bb6d-5099288a42e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:01:55.205795 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.205760    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:55.205795 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.205789    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:55.205993 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.205807    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:55.205993 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.205821    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:55.205993 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.205834    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:55.206969 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.206937    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-envoy" pod "90a79cd0-bb0f-40a4-bb6d-5099288a42e8" (UID: "90a79cd0-bb0f-40a4-bb6d-5099288a42e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:01:55.207291 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.207266    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-podinfo" pod "90a79cd0-bb0f-40a4-bb6d-5099288a42e8" (UID: "90a79cd0-bb0f-40a4-bb6d-5099288a42e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:01:55.207363 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.207314    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-token" pod "90a79cd0-bb0f-40a4-bb6d-5099288a42e8" (UID: "90a79cd0-bb0f-40a4-bb6d-5099288a42e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:01:55.207456 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.207440    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-kube-api-access-9mlhl" pod "90a79cd0-bb0f-40a4-bb6d-5099288a42e8" (UID: "90a79cd0-bb0f-40a4-bb6d-5099288a42e8"). InnerVolumeSpecName "kube-api-access-9mlhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:01:55.307457 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.307427    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:55.307457 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.307455    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:55.307457 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.307463    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:55.307678 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.307472    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mlhl\" (UniqueName: \"kubernetes.io/projected/90a79cd0-bb0f-40a4-bb6d-5099288a42e8-kube-api-access-9mlhl\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:01:55.610491 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:01:55.610449    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:01:55.945337 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.945230    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"
Oct 30 05:01:55.945337 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.945278    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4" event={"ID":"90a79cd0-bb0f-40a4-bb6d-5099288a42e8","Type":"ContainerDied","Data":"3077718e1e612449b618debe995c9856305078a0f1eb23d42b43b272f32b69d8"}
Oct 30 05:01:55.945337 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.945338    4874 scope.go:122] "RemoveContainer" containerID="78b2a4b3b1484fad04de8b1d0911a6b074331c3049cdf98f47ce918f93cd49ba"
Oct 30 05:01:55.996712 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.996674    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"]
Oct 30 05:01:55.999847 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:55.999824    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-lxzs4"]
Oct 30 05:01:56.733288 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:01:56.733257    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90a79cd0-bb0f-40a4-bb6d-5099288a42e8" path="/var/lib/kubelet/pods/90a79cd0-bb0f-40a4-bb6d-5099288a42e8/volumes"
Oct 30 05:02:01.744542 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:01.744510    4874 scope.go:122] "RemoveContainer" containerID="b4dd3c7c14e3397330f15c411b606d08d74b7110247abfd86d391d229d807f4d"
Oct 30 05:02:08.114640 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:08.114598    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg"]
Oct 30 05:02:08.115070 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:08.114849    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg" podUID="cd62bd73-c45b-465d-b3e5-93b0bf16570a" containerName="maas-api" containerID="cri-o://d78f3469c270a2ebf1dac3a3cd0d40dc8b07cc0c2f63f4eefdaae711241f86d8" gracePeriod=30
Oct 30 05:02:08.479743 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:08.479712    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg"
Oct 30 05:02:08.583306 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:08.563896    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd62bd73-c45b-465d-b3e5-93b0bf16570a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd62bd73-c45b-465d-b3e5-93b0bf16570a-maas-api-tls\") pod \"cd62bd73-c45b-465d-b3e5-93b0bf16570a\" (UID: \"cd62bd73-c45b-465d-b3e5-93b0bf16570a\") "
Oct 30 05:02:08.583306 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:08.564108    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd62bd73-c45b-465d-b3e5-93b0bf16570a-kube-api-access-hwwxf\" (UniqueName: \"kubernetes.io/projected/cd62bd73-c45b-465d-b3e5-93b0bf16570a-kube-api-access-hwwxf\") pod \"cd62bd73-c45b-465d-b3e5-93b0bf16570a\" (UID: \"cd62bd73-c45b-465d-b3e5-93b0bf16570a\") "
Oct 30 05:02:08.583306 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:08.564273    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd62bd73-c45b-465d-b3e5-93b0bf16570a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd62bd73-c45b-465d-b3e5-93b0bf16570a-metrics-certs\") pod \"cd62bd73-c45b-465d-b3e5-93b0bf16570a\" (UID: \"cd62bd73-c45b-465d-b3e5-93b0bf16570a\") "
Oct 30 05:02:08.583306 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:08.564402    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd62bd73-c45b-465d-b3e5-93b0bf16570a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd62bd73-c45b-465d-b3e5-93b0bf16570a-openshift-service-ca\") pod \"cd62bd73-c45b-465d-b3e5-93b0bf16570a\" (UID: \"cd62bd73-c45b-465d-b3e5-93b0bf16570a\") "
Oct 30 05:02:08.583306 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:08.569363    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd62bd73-c45b-465d-b3e5-93b0bf16570a-openshift-service-ca" pod "cd62bd73-c45b-465d-b3e5-93b0bf16570a" (UID: "cd62bd73-c45b-465d-b3e5-93b0bf16570a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:02:08.599533 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:08.599487    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd62bd73-c45b-465d-b3e5-93b0bf16570a-kube-api-access-hwwxf" pod "cd62bd73-c45b-465d-b3e5-93b0bf16570a" (UID: "cd62bd73-c45b-465d-b3e5-93b0bf16570a"). InnerVolumeSpecName "kube-api-access-hwwxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:02:08.638861 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:08.638800    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd62bd73-c45b-465d-b3e5-93b0bf16570a-metrics-certs" pod "cd62bd73-c45b-465d-b3e5-93b0bf16570a" (UID: "cd62bd73-c45b-465d-b3e5-93b0bf16570a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:02:08.658326 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:08.658261    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd62bd73-c45b-465d-b3e5-93b0bf16570a-maas-api-tls" pod "cd62bd73-c45b-465d-b3e5-93b0bf16570a" (UID: "cd62bd73-c45b-465d-b3e5-93b0bf16570a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:02:08.675026 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:08.674916    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwwxf\" (UniqueName: \"kubernetes.io/projected/cd62bd73-c45b-465d-b3e5-93b0bf16570a-kube-api-access-hwwxf\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:02:08.675026 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:08.674964    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd62bd73-c45b-465d-b3e5-93b0bf16570a-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:02:08.675026 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:08.674980    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd62bd73-c45b-465d-b3e5-93b0bf16570a-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:02:08.675026 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:08.674996    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd62bd73-c45b-465d-b3e5-93b0bf16570a-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:02:09.056375 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:09.056287    4874 generic.go:356] "Generic (PLEG): container finished" podID="cd62bd73-c45b-465d-b3e5-93b0bf16570a" containerID="d78f3469c270a2ebf1dac3a3cd0d40dc8b07cc0c2f63f4eefdaae711241f86d8" exitCode=0
Oct 30 05:02:09.056375 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:09.056349    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg" event={"ID":"cd62bd73-c45b-465d-b3e5-93b0bf16570a","Type":"ContainerDied","Data":"d78f3469c270a2ebf1dac3a3cd0d40dc8b07cc0c2f63f4eefdaae711241f86d8"}
Oct 30 05:02:09.056604 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:09.056390    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg" event={"ID":"cd62bd73-c45b-465d-b3e5-93b0bf16570a","Type":"ContainerDied","Data":"2b0653a075b2acb5889196295aff89795e2152519eb8e8a7c6ea4b2985486b22"}
Oct 30 05:02:09.056604 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:09.056411    4874 scope.go:122] "RemoveContainer" containerID="d78f3469c270a2ebf1dac3a3cd0d40dc8b07cc0c2f63f4eefdaae711241f86d8"
Oct 30 05:02:09.056604 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:09.056539    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg"
Oct 30 05:02:09.068604 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:09.068585    4874 scope.go:122] "RemoveContainer" containerID="d78f3469c270a2ebf1dac3a3cd0d40dc8b07cc0c2f63f4eefdaae711241f86d8"
Oct 30 05:02:09.068872 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:02:09.068853    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d78f3469c270a2ebf1dac3a3cd0d40dc8b07cc0c2f63f4eefdaae711241f86d8\": container with ID starting with d78f3469c270a2ebf1dac3a3cd0d40dc8b07cc0c2f63f4eefdaae711241f86d8 not found: ID does not exist" containerID="d78f3469c270a2ebf1dac3a3cd0d40dc8b07cc0c2f63f4eefdaae711241f86d8"
Oct 30 05:02:09.068922 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:09.068880    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d78f3469c270a2ebf1dac3a3cd0d40dc8b07cc0c2f63f4eefdaae711241f86d8"} err="failed to get container status \"d78f3469c270a2ebf1dac3a3cd0d40dc8b07cc0c2f63f4eefdaae711241f86d8\": rpc error: code = NotFound desc = could not find container \"d78f3469c270a2ebf1dac3a3cd0d40dc8b07cc0c2f63f4eefdaae711241f86d8\": container with ID starting with d78f3469c270a2ebf1dac3a3cd0d40dc8b07cc0c2f63f4eefdaae711241f86d8 not found: ID does not exist"
Oct 30 05:02:09.107209 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:09.107182    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg"]
Oct 30 05:02:09.111026 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:09.111004    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-134dad-548684db4c-7b5xg"]
Oct 30 05:02:10.731721 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:10.731690    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd62bd73-c45b-465d-b3e5-93b0bf16570a" path="/var/lib/kubelet/pods/cd62bd73-c45b-465d-b3e5-93b0bf16570a/volumes"
Oct 30 05:02:14.370730 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:14.370686    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:02:25.610573 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:02:25.610536    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:02:40.010702 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:40.010666    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"]
Oct 30 05:02:40.011190 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:40.010978    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7" podUID="f33dee17-9ac8-4ea0-b547-94b24e63b4bb" containerName="istio-proxy" containerID="cri-o://95d6e45010658a5cf414a1beb6294a30258481f8445b7172f192d6d93d6c424f" gracePeriod=30
Oct 30 05:02:42.566120 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:42.566082    4874 patch_prober.go:29] interesting pod/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:02:42.566607 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:42.566156    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7" podUID="f33dee17-9ac8-4ea0-b547-94b24e63b4bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:02:44.699767 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:44.699723    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"]
Oct 30 05:02:44.700298 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:44.700098    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb" podUID="06eecd27-740b-48d7-903c-182db0dae7ad" containerName="istio-proxy" containerID="cri-o://0e71fb7f3a0f1f044214c99ced0c32571bc809c6f406d9bc7eadeaeee199e806" gracePeriod=30
Oct 30 05:02:45.284677 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.284618    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 05:02:45.376395 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.376345    4874 generic.go:356] "Generic (PLEG): container finished" podID="f33dee17-9ac8-4ea0-b547-94b24e63b4bb" containerID="95d6e45010658a5cf414a1beb6294a30258481f8445b7172f192d6d93d6c424f" exitCode=0
Oct 30 05:02:45.376556 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.376415    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"
Oct 30 05:02:45.376556 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.376413    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7" event={"ID":"f33dee17-9ac8-4ea0-b547-94b24e63b4bb","Type":"ContainerDied","Data":"95d6e45010658a5cf414a1beb6294a30258481f8445b7172f192d6d93d6c424f"}
Oct 30 05:02:45.376645 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.376553    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7" event={"ID":"f33dee17-9ac8-4ea0-b547-94b24e63b4bb","Type":"ContainerDied","Data":"6ed4e92c927a28052af9bba06278e93a0575de8f0ec00425c5f9ea67ea22f7e7"}
Oct 30 05:02:45.376645 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.376588    4874 scope.go:122] "RemoveContainer" containerID="95d6e45010658a5cf414a1beb6294a30258481f8445b7172f192d6d93d6c424f"
Oct 30 05:02:45.390621 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.390429    4874 scope.go:122] "RemoveContainer" containerID="fa28a29695912d6deadec13216d0bf916eacf092ffc9c8a670da8e5ed54b49a9"
Oct 30 05:02:45.405348 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.405327    4874 scope.go:122] "RemoveContainer" containerID="95d6e45010658a5cf414a1beb6294a30258481f8445b7172f192d6d93d6c424f"
Oct 30 05:02:45.405676 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:02:45.405646    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95d6e45010658a5cf414a1beb6294a30258481f8445b7172f192d6d93d6c424f\": container with ID starting with 95d6e45010658a5cf414a1beb6294a30258481f8445b7172f192d6d93d6c424f not found: ID does not exist" containerID="95d6e45010658a5cf414a1beb6294a30258481f8445b7172f192d6d93d6c424f"
Oct 30 05:02:45.405795 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.405687    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d6e45010658a5cf414a1beb6294a30258481f8445b7172f192d6d93d6c424f"} err="failed to get container status \"95d6e45010658a5cf414a1beb6294a30258481f8445b7172f192d6d93d6c424f\": rpc error: code = NotFound desc = could not find container \"95d6e45010658a5cf414a1beb6294a30258481f8445b7172f192d6d93d6c424f\": container with ID starting with 95d6e45010658a5cf414a1beb6294a30258481f8445b7172f192d6d93d6c424f not found: ID does not exist"
Oct 30 05:02:45.405795 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.405712    4874 scope.go:122] "RemoveContainer" containerID="fa28a29695912d6deadec13216d0bf916eacf092ffc9c8a670da8e5ed54b49a9"
Oct 30 05:02:45.406056 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:02:45.406030    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa28a29695912d6deadec13216d0bf916eacf092ffc9c8a670da8e5ed54b49a9\": container with ID starting with fa28a29695912d6deadec13216d0bf916eacf092ffc9c8a670da8e5ed54b49a9 not found: ID does not exist" containerID="fa28a29695912d6deadec13216d0bf916eacf092ffc9c8a670da8e5ed54b49a9"
Oct 30 05:02:45.406132 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.406067    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa28a29695912d6deadec13216d0bf916eacf092ffc9c8a670da8e5ed54b49a9"} err="failed to get container status \"fa28a29695912d6deadec13216d0bf916eacf092ffc9c8a670da8e5ed54b49a9\": rpc error: code = NotFound desc = could not find container \"fa28a29695912d6deadec13216d0bf916eacf092ffc9c8a670da8e5ed54b49a9\": container with ID starting with fa28a29695912d6deadec13216d0bf916eacf092ffc9c8a670da8e5ed54b49a9 not found: ID does not exist"
Oct 30 05:02:45.431346 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.431321    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-workload-certs\") pod \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") "
Oct 30 05:02:45.431472 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.431366    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-envoy\") pod \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") "
Oct 30 05:02:45.431472 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.431400    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-credential-socket\") pod \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") "
Oct 30 05:02:45.431472 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.431429    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-podinfo\") pod \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") "
Oct 30 05:02:45.431632 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.431579    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-data\") pod \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") "
Oct 30 05:02:45.431692 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.431625    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-workload-certs" pod "f33dee17-9ac8-4ea0-b547-94b24e63b4bb" (UID: "f33dee17-9ac8-4ea0-b547-94b24e63b4bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:02:45.431692 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.431655    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-credential-socket" pod "f33dee17-9ac8-4ea0-b547-94b24e63b4bb" (UID: "f33dee17-9ac8-4ea0-b547-94b24e63b4bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:02:45.431692 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.431663    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-workload-socket\") pod \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") "
Oct 30 05:02:45.431850 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.431821    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-data" pod "f33dee17-9ac8-4ea0-b547-94b24e63b4bb" (UID: "f33dee17-9ac8-4ea0-b547-94b24e63b4bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:02:45.431912 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.431823    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istiod-ca-cert\") pod \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") "
Oct 30 05:02:45.431912 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.431862    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-workload-socket" pod "f33dee17-9ac8-4ea0-b547-94b24e63b4bb" (UID: "f33dee17-9ac8-4ea0-b547-94b24e63b4bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:02:45.431912 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.431900    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-kube-api-access-g2gp8\" (UniqueName: \"kubernetes.io/projected/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-kube-api-access-g2gp8\") pod \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") "
Oct 30 05:02:45.432068 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.431936    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-token\" (UniqueName: \"kubernetes.io/projected/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-token\") pod \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\" (UID: \"f33dee17-9ac8-4ea0-b547-94b24e63b4bb\") "
Oct 30 05:02:45.432122 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.432096    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istiod-ca-cert" pod "f33dee17-9ac8-4ea0-b547-94b24e63b4bb" (UID: "f33dee17-9ac8-4ea0-b547-94b24e63b4bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:02:45.432666 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.432641    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:02:45.432789 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.432669    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:02:45.432789 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.432684    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:02:45.432789 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.432699    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:02:45.432789 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.432712    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:02:45.433817 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.433791    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-envoy" pod "f33dee17-9ac8-4ea0-b547-94b24e63b4bb" (UID: "f33dee17-9ac8-4ea0-b547-94b24e63b4bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:02:45.433906 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.433840    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-podinfo" pod "f33dee17-9ac8-4ea0-b547-94b24e63b4bb" (UID: "f33dee17-9ac8-4ea0-b547-94b24e63b4bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:02:45.434298 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.434234    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-kube-api-access-g2gp8" pod "f33dee17-9ac8-4ea0-b547-94b24e63b4bb" (UID: "f33dee17-9ac8-4ea0-b547-94b24e63b4bb"). InnerVolumeSpecName "kube-api-access-g2gp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:02:45.434396 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.434360    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-token" pod "f33dee17-9ac8-4ea0-b547-94b24e63b4bb" (UID: "f33dee17-9ac8-4ea0-b547-94b24e63b4bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:02:45.533560 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.533524    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2gp8\" (UniqueName: \"kubernetes.io/projected/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-kube-api-access-g2gp8\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:02:45.533560 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.533553    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:02:45.533560 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.533565    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:02:45.533780 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.533576    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f33dee17-9ac8-4ea0-b547-94b24e63b4bb-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:02:45.721966 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.721931    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"]
Oct 30 05:02:45.734156 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:45.734123    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5f1508-openshift-default-6669f97b5-5dsb7"]
Oct 30 05:02:46.731864 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:46.731833    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f33dee17-9ac8-4ea0-b547-94b24e63b4bb" path="/var/lib/kubelet/pods/f33dee17-9ac8-4ea0-b547-94b24e63b4bb/volumes"
Oct 30 05:02:49.992147 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:49.992122    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 05:02:50.005354 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.005330    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06eecd27-740b-48d7-903c-182db0dae7ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06eecd27-740b-48d7-903c-182db0dae7ad-istiod-ca-cert\") pod \"06eecd27-740b-48d7-903c-182db0dae7ad\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") "
Oct 30 05:02:50.005490 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.005472    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-workload-certs\") pod \"06eecd27-740b-48d7-903c-182db0dae7ad\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") "
Oct 30 05:02:50.005620 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.005602    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-istio-envoy\") pod \"06eecd27-740b-48d7-903c-182db0dae7ad\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") "
Oct 30 05:02:50.005871 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.005690    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06eecd27-740b-48d7-903c-182db0dae7ad-kube-api-access-mbkhl\" (UniqueName: \"kubernetes.io/projected/06eecd27-740b-48d7-903c-182db0dae7ad-kube-api-access-mbkhl\") pod \"06eecd27-740b-48d7-903c-182db0dae7ad\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") "
Oct 30 05:02:50.005871 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.005737    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-workload-socket\") pod \"06eecd27-740b-48d7-903c-182db0dae7ad\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") "
Oct 30 05:02:50.005871 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.005782    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06eecd27-740b-48d7-903c-182db0dae7ad-istio-token\" (UniqueName: \"kubernetes.io/projected/06eecd27-740b-48d7-903c-182db0dae7ad-istio-token\") pod \"06eecd27-740b-48d7-903c-182db0dae7ad\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") "
Oct 30 05:02:50.005871 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.005843    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-credential-socket\") pod \"06eecd27-740b-48d7-903c-182db0dae7ad\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") "
Oct 30 05:02:50.006100 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.005879    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-istio-data\") pod \"06eecd27-740b-48d7-903c-182db0dae7ad\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") "
Oct 30 05:02:50.006100 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.005917    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06eecd27-740b-48d7-903c-182db0dae7ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06eecd27-740b-48d7-903c-182db0dae7ad-istio-podinfo\") pod \"06eecd27-740b-48d7-903c-182db0dae7ad\" (UID: \"06eecd27-740b-48d7-903c-182db0dae7ad\") "
Oct 30 05:02:50.006100 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.005776    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-workload-certs" pod "06eecd27-740b-48d7-903c-182db0dae7ad" (UID: "06eecd27-740b-48d7-903c-182db0dae7ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:02:50.006100 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.005780    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06eecd27-740b-48d7-903c-182db0dae7ad-istiod-ca-cert" pod "06eecd27-740b-48d7-903c-182db0dae7ad" (UID: "06eecd27-740b-48d7-903c-182db0dae7ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:02:50.006100 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.005993    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-workload-socket" pod "06eecd27-740b-48d7-903c-182db0dae7ad" (UID: "06eecd27-740b-48d7-903c-182db0dae7ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:02:50.006442 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.006126    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-credential-socket" pod "06eecd27-740b-48d7-903c-182db0dae7ad" (UID: "06eecd27-740b-48d7-903c-182db0dae7ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:02:50.006517 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.006457    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-istio-data" pod "06eecd27-740b-48d7-903c-182db0dae7ad" (UID: "06eecd27-740b-48d7-903c-182db0dae7ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:02:50.006682 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.006658    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:02:50.006795 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.006685    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:02:50.006795 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.006700    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06eecd27-740b-48d7-903c-182db0dae7ad-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:02:50.006795 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.006715    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:02:50.006795 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.006729    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:02:50.007988 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.007961    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-istio-envoy" pod "06eecd27-740b-48d7-903c-182db0dae7ad" (UID: "06eecd27-740b-48d7-903c-182db0dae7ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:02:50.008075 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.008028    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06eecd27-740b-48d7-903c-182db0dae7ad-kube-api-access-mbkhl" pod "06eecd27-740b-48d7-903c-182db0dae7ad" (UID: "06eecd27-740b-48d7-903c-182db0dae7ad"). InnerVolumeSpecName "kube-api-access-mbkhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:02:50.008075 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.008046    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06eecd27-740b-48d7-903c-182db0dae7ad-istio-token" pod "06eecd27-740b-48d7-903c-182db0dae7ad" (UID: "06eecd27-740b-48d7-903c-182db0dae7ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:02:50.008192 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.008107    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06eecd27-740b-48d7-903c-182db0dae7ad-istio-podinfo" pod "06eecd27-740b-48d7-903c-182db0dae7ad" (UID: "06eecd27-740b-48d7-903c-182db0dae7ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:02:50.108001 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.107972    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06eecd27-740b-48d7-903c-182db0dae7ad-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:02:50.108001 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.107998    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbkhl\" (UniqueName: \"kubernetes.io/projected/06eecd27-740b-48d7-903c-182db0dae7ad-kube-api-access-mbkhl\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:02:50.108001 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.108009    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06eecd27-740b-48d7-903c-182db0dae7ad-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:02:50.108231 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.108018    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06eecd27-740b-48d7-903c-182db0dae7ad-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:02:50.422127 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.422093    4874 generic.go:356] "Generic (PLEG): container finished" podID="06eecd27-740b-48d7-903c-182db0dae7ad" containerID="0e71fb7f3a0f1f044214c99ced0c32571bc809c6f406d9bc7eadeaeee199e806" exitCode=0
Oct 30 05:02:50.422335 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.422140    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"
Oct 30 05:02:50.422335 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.422168    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb" event={"ID":"06eecd27-740b-48d7-903c-182db0dae7ad","Type":"ContainerDied","Data":"0e71fb7f3a0f1f044214c99ced0c32571bc809c6f406d9bc7eadeaeee199e806"}
Oct 30 05:02:50.422335 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.422221    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb" event={"ID":"06eecd27-740b-48d7-903c-182db0dae7ad","Type":"ContainerDied","Data":"62f3db08928d003e7170f0c3d45e05706e4f4cf2e3437768c1f4238e356245ed"}
Oct 30 05:02:50.422335 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.422278    4874 scope.go:122] "RemoveContainer" containerID="0e71fb7f3a0f1f044214c99ced0c32571bc809c6f406d9bc7eadeaeee199e806"
Oct 30 05:02:50.433841 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.433823    4874 scope.go:122] "RemoveContainer" containerID="8163ae0c23b5fbf2851e182dbbad87efd76e629599cfc1623582f8c765610220"
Oct 30 05:02:50.443362 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.443346    4874 scope.go:122] "RemoveContainer" containerID="0e71fb7f3a0f1f044214c99ced0c32571bc809c6f406d9bc7eadeaeee199e806"
Oct 30 05:02:50.443613 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:02:50.443595    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e71fb7f3a0f1f044214c99ced0c32571bc809c6f406d9bc7eadeaeee199e806\": container with ID starting with 0e71fb7f3a0f1f044214c99ced0c32571bc809c6f406d9bc7eadeaeee199e806 not found: ID does not exist" containerID="0e71fb7f3a0f1f044214c99ced0c32571bc809c6f406d9bc7eadeaeee199e806"
Oct 30 05:02:50.443662 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.443631    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e71fb7f3a0f1f044214c99ced0c32571bc809c6f406d9bc7eadeaeee199e806"} err="failed to get container status \"0e71fb7f3a0f1f044214c99ced0c32571bc809c6f406d9bc7eadeaeee199e806\": rpc error: code = NotFound desc = could not find container \"0e71fb7f3a0f1f044214c99ced0c32571bc809c6f406d9bc7eadeaeee199e806\": container with ID starting with 0e71fb7f3a0f1f044214c99ced0c32571bc809c6f406d9bc7eadeaeee199e806 not found: ID does not exist"
Oct 30 05:02:50.443662 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.443648    4874 scope.go:122] "RemoveContainer" containerID="8163ae0c23b5fbf2851e182dbbad87efd76e629599cfc1623582f8c765610220"
Oct 30 05:02:50.443854 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:02:50.443837    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8163ae0c23b5fbf2851e182dbbad87efd76e629599cfc1623582f8c765610220\": container with ID starting with 8163ae0c23b5fbf2851e182dbbad87efd76e629599cfc1623582f8c765610220 not found: ID does not exist" containerID="8163ae0c23b5fbf2851e182dbbad87efd76e629599cfc1623582f8c765610220"
Oct 30 05:02:50.443896 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.443861    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8163ae0c23b5fbf2851e182dbbad87efd76e629599cfc1623582f8c765610220"} err="failed to get container status \"8163ae0c23b5fbf2851e182dbbad87efd76e629599cfc1623582f8c765610220\": rpc error: code = NotFound desc = could not find container \"8163ae0c23b5fbf2851e182dbbad87efd76e629599cfc1623582f8c765610220\": container with ID starting with 8163ae0c23b5fbf2851e182dbbad87efd76e629599cfc1623582f8c765610220 not found: ID does not exist"
Oct 30 05:02:50.472742 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.472721    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"]
Oct 30 05:02:50.484104 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.484083    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-gq7jb"]
Oct 30 05:02:50.732174 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:02:50.732138    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06eecd27-740b-48d7-903c-182db0dae7ad" path="/var/lib/kubelet/pods/06eecd27-740b-48d7-903c-182db0dae7ad/volumes"
Oct 30 05:02:55.610887 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:02:55.610840    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:03:01.293720 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:01.293690    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 05:03:01.295969 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:01.295948    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 05:03:13.538467 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:13.538373    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz"]
Oct 30 05:03:13.538939 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:13.538601    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz" podUID="2b1ce86c-0464-4511-bed6-acefb0d3b841" containerName="maas-api" containerID="cri-o://0dd75d783a8524acfad44c4211bc04fd0c85cbad987b6d80c080cbae5e55e785" gracePeriod=30
Oct 30 05:03:13.889499 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:13.889474    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz"
Oct 30 05:03:13.953265 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:13.947853    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b1ce86c-0464-4511-bed6-acefb0d3b841-kube-api-access-g79sc\" (UniqueName: \"kubernetes.io/projected/2b1ce86c-0464-4511-bed6-acefb0d3b841-kube-api-access-g79sc\") pod \"2b1ce86c-0464-4511-bed6-acefb0d3b841\" (UID: \"2b1ce86c-0464-4511-bed6-acefb0d3b841\") "
Oct 30 05:03:13.953265 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:13.947999    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b1ce86c-0464-4511-bed6-acefb0d3b841-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b1ce86c-0464-4511-bed6-acefb0d3b841-maas-api-tls\") pod \"2b1ce86c-0464-4511-bed6-acefb0d3b841\" (UID: \"2b1ce86c-0464-4511-bed6-acefb0d3b841\") "
Oct 30 05:03:13.953265 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:13.948037    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b1ce86c-0464-4511-bed6-acefb0d3b841-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b1ce86c-0464-4511-bed6-acefb0d3b841-metrics-certs\") pod \"2b1ce86c-0464-4511-bed6-acefb0d3b841\" (UID: \"2b1ce86c-0464-4511-bed6-acefb0d3b841\") "
Oct 30 05:03:13.953265 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:13.948117    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b1ce86c-0464-4511-bed6-acefb0d3b841-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b1ce86c-0464-4511-bed6-acefb0d3b841-openshift-service-ca\") pod \"2b1ce86c-0464-4511-bed6-acefb0d3b841\" (UID: \"2b1ce86c-0464-4511-bed6-acefb0d3b841\") "
Oct 30 05:03:13.981499 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:13.978226    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b1ce86c-0464-4511-bed6-acefb0d3b841-openshift-service-ca" pod "2b1ce86c-0464-4511-bed6-acefb0d3b841" (UID: "2b1ce86c-0464-4511-bed6-acefb0d3b841"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:03:13.985262 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:13.983631    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1ce86c-0464-4511-bed6-acefb0d3b841-kube-api-access-g79sc" pod "2b1ce86c-0464-4511-bed6-acefb0d3b841" (UID: "2b1ce86c-0464-4511-bed6-acefb0d3b841"). InnerVolumeSpecName "kube-api-access-g79sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:03:13.993258 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:13.988463    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b1ce86c-0464-4511-bed6-acefb0d3b841-maas-api-tls" pod "2b1ce86c-0464-4511-bed6-acefb0d3b841" (UID: "2b1ce86c-0464-4511-bed6-acefb0d3b841"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:03:14.027085 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:14.024444    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b1ce86c-0464-4511-bed6-acefb0d3b841-metrics-certs" pod "2b1ce86c-0464-4511-bed6-acefb0d3b841" (UID: "2b1ce86c-0464-4511-bed6-acefb0d3b841"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:03:14.061438 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:14.061383    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b1ce86c-0464-4511-bed6-acefb0d3b841-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:03:14.061438 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:14.061419    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b1ce86c-0464-4511-bed6-acefb0d3b841-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:03:14.061438 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:14.061431    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b1ce86c-0464-4511-bed6-acefb0d3b841-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:03:14.061438 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:14.061445    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g79sc\" (UniqueName: \"kubernetes.io/projected/2b1ce86c-0464-4511-bed6-acefb0d3b841-kube-api-access-g79sc\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:03:14.581901 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:14.581870    4874 generic.go:356] "Generic (PLEG): container finished" podID="2b1ce86c-0464-4511-bed6-acefb0d3b841" containerID="0dd75d783a8524acfad44c4211bc04fd0c85cbad987b6d80c080cbae5e55e785" exitCode=0
Oct 30 05:03:14.582394 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:14.581919    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz"
Oct 30 05:03:14.582394 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:14.581954    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz" event={"ID":"2b1ce86c-0464-4511-bed6-acefb0d3b841","Type":"ContainerDied","Data":"0dd75d783a8524acfad44c4211bc04fd0c85cbad987b6d80c080cbae5e55e785"}
Oct 30 05:03:14.582394 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:14.582007    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz" event={"ID":"2b1ce86c-0464-4511-bed6-acefb0d3b841","Type":"ContainerDied","Data":"e18858371dfe5701fb4d27e2ab2d024a514f2985fab05ceda2461c43256c15ae"}
Oct 30 05:03:14.582394 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:14.582030    4874 scope.go:122] "RemoveContainer" containerID="0dd75d783a8524acfad44c4211bc04fd0c85cbad987b6d80c080cbae5e55e785"
Oct 30 05:03:14.592580 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:14.592561    4874 scope.go:122] "RemoveContainer" containerID="0dd75d783a8524acfad44c4211bc04fd0c85cbad987b6d80c080cbae5e55e785"
Oct 30 05:03:14.592845 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:03:14.592813    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dd75d783a8524acfad44c4211bc04fd0c85cbad987b6d80c080cbae5e55e785\": container with ID starting with 0dd75d783a8524acfad44c4211bc04fd0c85cbad987b6d80c080cbae5e55e785 not found: ID does not exist" containerID="0dd75d783a8524acfad44c4211bc04fd0c85cbad987b6d80c080cbae5e55e785"
Oct 30 05:03:14.592898 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:14.592852    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dd75d783a8524acfad44c4211bc04fd0c85cbad987b6d80c080cbae5e55e785"} err="failed to get container status \"0dd75d783a8524acfad44c4211bc04fd0c85cbad987b6d80c080cbae5e55e785\": rpc error: code = NotFound desc = could not find container \"0dd75d783a8524acfad44c4211bc04fd0c85cbad987b6d80c080cbae5e55e785\": container with ID starting with 0dd75d783a8524acfad44c4211bc04fd0c85cbad987b6d80c080cbae5e55e785 not found: ID does not exist"
Oct 30 05:03:14.632173 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:14.632147    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz"]
Oct 30 05:03:14.643468 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:14.643449    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-82bec5-9f58c4f86-hhtzz"]
Oct 30 05:03:14.730907 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:14.730874    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b1ce86c-0464-4511-bed6-acefb0d3b841" path="/var/lib/kubelet/pods/2b1ce86c-0464-4511-bed6-acefb0d3b841/volumes"
Oct 30 05:03:23.134002 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:23.133970    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"]
Oct 30 05:03:23.134526 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:23.134155    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8" podUID="b35dc9fb-6f3c-4425-8360-6f52c94f1909" containerName="istio-proxy" containerID="cri-o://3d74768d638f2a73fd7dc15ab7f55f9568afa3c3c80e5b48cb887b8269fd133c" gracePeriod=30
Oct 30 05:03:25.206509 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:25.206475    4874 patch_prober.go:29] interesting pod/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:03:25.206972 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:25.206533    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8" podUID="b35dc9fb-6f3c-4425-8360-6f52c94f1909" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:03:25.609660 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:03:25.609624    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:03:28.440864 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.440830    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 05:03:28.573439 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.573404    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-podinfo\") pod \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") "
Oct 30 05:03:28.573638 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.573483    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b35dc9fb-6f3c-4425-8360-6f52c94f1909-kube-api-access-c6spg\" (UniqueName: \"kubernetes.io/projected/b35dc9fb-6f3c-4425-8360-6f52c94f1909-kube-api-access-c6spg\") pod \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") "
Oct 30 05:03:28.573638 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.573565    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-envoy\") pod \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") "
Oct 30 05:03:28.573769 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.573675    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istiod-ca-cert\") pod \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") "
Oct 30 05:03:28.573769 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.573757    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-workload-certs\") pod \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") "
Oct 30 05:03:28.573954 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.573805    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-workload-socket\") pod \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") "
Oct 30 05:03:28.573954 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.573842    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-token\" (UniqueName: \"kubernetes.io/projected/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-token\") pod \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") "
Oct 30 05:03:28.573954 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.573944    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-credential-socket\") pod \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") "
Oct 30 05:03:28.574120 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.573991    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-data\") pod \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\" (UID: \"b35dc9fb-6f3c-4425-8360-6f52c94f1909\") "
Oct 30 05:03:28.574180 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.574156    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istiod-ca-cert" pod "b35dc9fb-6f3c-4425-8360-6f52c94f1909" (UID: "b35dc9fb-6f3c-4425-8360-6f52c94f1909"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:03:28.574541 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.574498    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-workload-certs" pod "b35dc9fb-6f3c-4425-8360-6f52c94f1909" (UID: "b35dc9fb-6f3c-4425-8360-6f52c94f1909"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:03:28.574695 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.574652    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-credential-socket" pod "b35dc9fb-6f3c-4425-8360-6f52c94f1909" (UID: "b35dc9fb-6f3c-4425-8360-6f52c94f1909"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:03:28.574695 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.574668    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-data" pod "b35dc9fb-6f3c-4425-8360-6f52c94f1909" (UID: "b35dc9fb-6f3c-4425-8360-6f52c94f1909"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:03:28.574828 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.574786    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-workload-socket" pod "b35dc9fb-6f3c-4425-8360-6f52c94f1909" (UID: "b35dc9fb-6f3c-4425-8360-6f52c94f1909"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:03:28.575304 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.575204    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:03:28.575304 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.575229    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:03:28.575304 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.575256    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:03:28.575304 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.575271    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:03:28.575304 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.575285    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:03:28.576875 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.576851    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-podinfo" pod "b35dc9fb-6f3c-4425-8360-6f52c94f1909" (UID: "b35dc9fb-6f3c-4425-8360-6f52c94f1909"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:03:28.576991 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.576894    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-envoy" pod "b35dc9fb-6f3c-4425-8360-6f52c94f1909" (UID: "b35dc9fb-6f3c-4425-8360-6f52c94f1909"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:03:28.576991 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.576909    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-token" pod "b35dc9fb-6f3c-4425-8360-6f52c94f1909" (UID: "b35dc9fb-6f3c-4425-8360-6f52c94f1909"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:03:28.577222 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.577205    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b35dc9fb-6f3c-4425-8360-6f52c94f1909-kube-api-access-c6spg" pod "b35dc9fb-6f3c-4425-8360-6f52c94f1909" (UID: "b35dc9fb-6f3c-4425-8360-6f52c94f1909"). InnerVolumeSpecName "kube-api-access-c6spg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:03:28.676225 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.676170    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:03:28.676225 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.676192    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6spg\" (UniqueName: \"kubernetes.io/projected/b35dc9fb-6f3c-4425-8360-6f52c94f1909-kube-api-access-c6spg\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:03:28.676225 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.676206    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:03:28.676225 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.676218    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b35dc9fb-6f3c-4425-8360-6f52c94f1909-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:03:28.700432 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.700408    4874 generic.go:356] "Generic (PLEG): container finished" podID="b35dc9fb-6f3c-4425-8360-6f52c94f1909" containerID="3d74768d638f2a73fd7dc15ab7f55f9568afa3c3c80e5b48cb887b8269fd133c" exitCode=0
Oct 30 05:03:28.700508 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.700453    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"
Oct 30 05:03:28.700508 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.700487    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8" event={"ID":"b35dc9fb-6f3c-4425-8360-6f52c94f1909","Type":"ContainerDied","Data":"3d74768d638f2a73fd7dc15ab7f55f9568afa3c3c80e5b48cb887b8269fd133c"}
Oct 30 05:03:28.700593 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.700536    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8" event={"ID":"b35dc9fb-6f3c-4425-8360-6f52c94f1909","Type":"ContainerDied","Data":"27241d7171c80562c248a453b74a51a448b12ceeb3abba26bb482c9118e62961"}
Oct 30 05:03:28.700593 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.700557    4874 scope.go:122] "RemoveContainer" containerID="3d74768d638f2a73fd7dc15ab7f55f9568afa3c3c80e5b48cb887b8269fd133c"
Oct 30 05:03:28.711619 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.711599    4874 scope.go:122] "RemoveContainer" containerID="4d7a9936c605f39601376b3421d7f3911c932dcd010bb07b11d456b28eceb6bd"
Oct 30 05:03:28.721123 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.721101    4874 scope.go:122] "RemoveContainer" containerID="3d74768d638f2a73fd7dc15ab7f55f9568afa3c3c80e5b48cb887b8269fd133c"
Oct 30 05:03:28.721409 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:03:28.721387    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d74768d638f2a73fd7dc15ab7f55f9568afa3c3c80e5b48cb887b8269fd133c\": container with ID starting with 3d74768d638f2a73fd7dc15ab7f55f9568afa3c3c80e5b48cb887b8269fd133c not found: ID does not exist" containerID="3d74768d638f2a73fd7dc15ab7f55f9568afa3c3c80e5b48cb887b8269fd133c"
Oct 30 05:03:28.721484 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.721417    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d74768d638f2a73fd7dc15ab7f55f9568afa3c3c80e5b48cb887b8269fd133c"} err="failed to get container status \"3d74768d638f2a73fd7dc15ab7f55f9568afa3c3c80e5b48cb887b8269fd133c\": rpc error: code = NotFound desc = could not find container \"3d74768d638f2a73fd7dc15ab7f55f9568afa3c3c80e5b48cb887b8269fd133c\": container with ID starting with 3d74768d638f2a73fd7dc15ab7f55f9568afa3c3c80e5b48cb887b8269fd133c not found: ID does not exist"
Oct 30 05:03:28.721484 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.721433    4874 scope.go:122] "RemoveContainer" containerID="4d7a9936c605f39601376b3421d7f3911c932dcd010bb07b11d456b28eceb6bd"
Oct 30 05:03:28.721666 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:03:28.721651    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d7a9936c605f39601376b3421d7f3911c932dcd010bb07b11d456b28eceb6bd\": container with ID starting with 4d7a9936c605f39601376b3421d7f3911c932dcd010bb07b11d456b28eceb6bd not found: ID does not exist" containerID="4d7a9936c605f39601376b3421d7f3911c932dcd010bb07b11d456b28eceb6bd"
Oct 30 05:03:28.721729 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.721670    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7a9936c605f39601376b3421d7f3911c932dcd010bb07b11d456b28eceb6bd"} err="failed to get container status \"4d7a9936c605f39601376b3421d7f3911c932dcd010bb07b11d456b28eceb6bd\": rpc error: code = NotFound desc = could not find container \"4d7a9936c605f39601376b3421d7f3911c932dcd010bb07b11d456b28eceb6bd\": container with ID starting with 4d7a9936c605f39601376b3421d7f3911c932dcd010bb07b11d456b28eceb6bd not found: ID does not exist"
Oct 30 05:03:28.751032 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.751007    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"]
Oct 30 05:03:28.756290 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:28.756268    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-hxqf8"]
Oct 30 05:03:30.729610 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:30.729578    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b35dc9fb-6f3c-4425-8360-6f52c94f1909" path="/var/lib/kubelet/pods/b35dc9fb-6f3c-4425-8360-6f52c94f1909/volumes"
Oct 30 05:03:37.171534 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:37.171501    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:03:52.086657 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:03:52.086624    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"]
Oct 30 05:03:55.610453 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:03:55.610408    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:04:07.107050 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.106994    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7" podUID="26c033bd-b0b9-4b75-ab44-b0ccc7895677" containerName="main" containerID="cri-o://f55b81f300a3e1e865c5a9d80a4f844352ce6c88d0f4aad0c74f3a38442044c6" gracePeriod=45
Oct 30 05:04:07.368077 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.368022    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 05:04:07.531960 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.531927    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-model-cache\") pod \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\" (UID: \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\") "
Oct 30 05:04:07.532154 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.531970    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-home\" (UniqueName: \"kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-home\") pod \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\" (UID: \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\") "
Oct 30 05:04:07.532154 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.532112    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-dshm\" (UniqueName: \"kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-dshm\") pod \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\" (UID: \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\") "
Oct 30 05:04:07.532154 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.532144    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26c033bd-b0b9-4b75-ab44-b0ccc7895677-tls-certs\" (UniqueName: \"kubernetes.io/secret/26c033bd-b0b9-4b75-ab44-b0ccc7895677-tls-certs\") pod \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\" (UID: \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\") "
Oct 30 05:04:07.532349 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.532175    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-model-cache" pod "26c033bd-b0b9-4b75-ab44-b0ccc7895677" (UID: "26c033bd-b0b9-4b75-ab44-b0ccc7895677"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:04:07.532349 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.532261    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-home" pod "26c033bd-b0b9-4b75-ab44-b0ccc7895677" (UID: "26c033bd-b0b9-4b75-ab44-b0ccc7895677"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:04:07.532465 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.532345    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26c033bd-b0b9-4b75-ab44-b0ccc7895677-kube-api-access-lg2ns\" (UniqueName: \"kubernetes.io/projected/26c033bd-b0b9-4b75-ab44-b0ccc7895677-kube-api-access-lg2ns\") pod \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\" (UID: \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\") "
Oct 30 05:04:07.532465 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.532384    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-tmp-dir\") pod \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\" (UID: \"26c033bd-b0b9-4b75-ab44-b0ccc7895677\") "
Oct 30 05:04:07.532689 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.532666    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-tmp-dir" pod "26c033bd-b0b9-4b75-ab44-b0ccc7895677" (UID: "26c033bd-b0b9-4b75-ab44-b0ccc7895677"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:04:07.532926 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.532909    4874 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-model-cache\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:04:07.532993 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.532931    4874 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-home\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:04:07.532993 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.532943    4874 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-tmp-dir\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:04:07.534417 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.534388    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-dshm" pod "26c033bd-b0b9-4b75-ab44-b0ccc7895677" (UID: "26c033bd-b0b9-4b75-ab44-b0ccc7895677"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:04:07.534523 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.534478    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c033bd-b0b9-4b75-ab44-b0ccc7895677-kube-api-access-lg2ns" pod "26c033bd-b0b9-4b75-ab44-b0ccc7895677" (UID: "26c033bd-b0b9-4b75-ab44-b0ccc7895677"). InnerVolumeSpecName "kube-api-access-lg2ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:04:07.534523 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.534490    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26c033bd-b0b9-4b75-ab44-b0ccc7895677-tls-certs" pod "26c033bd-b0b9-4b75-ab44-b0ccc7895677" (UID: "26c033bd-b0b9-4b75-ab44-b0ccc7895677"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:04:07.633842 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.633775    4874 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26c033bd-b0b9-4b75-ab44-b0ccc7895677-dshm\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:04:07.633842 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.633798    4874 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26c033bd-b0b9-4b75-ab44-b0ccc7895677-tls-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:04:07.633842 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.633809    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg2ns\" (UniqueName: \"kubernetes.io/projected/26c033bd-b0b9-4b75-ab44-b0ccc7895677-kube-api-access-lg2ns\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:04:07.985302 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.985269    4874 generic.go:356] "Generic (PLEG): container finished" podID="26c033bd-b0b9-4b75-ab44-b0ccc7895677" containerID="f55b81f300a3e1e865c5a9d80a4f844352ce6c88d0f4aad0c74f3a38442044c6" exitCode=0
Oct 30 05:04:07.985495 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.985321    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"
Oct 30 05:04:07.985495 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.985342    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7" event={"ID":"26c033bd-b0b9-4b75-ab44-b0ccc7895677","Type":"ContainerDied","Data":"f55b81f300a3e1e865c5a9d80a4f844352ce6c88d0f4aad0c74f3a38442044c6"}
Oct 30 05:04:07.985495 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.985381    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7" event={"ID":"26c033bd-b0b9-4b75-ab44-b0ccc7895677","Type":"ContainerDied","Data":"ad7ba30699f109e71b1923fba030fe6242e2bdcc0e5b35dfeda553b66c39f32f"}
Oct 30 05:04:07.985495 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.985398    4874 scope.go:122] "RemoveContainer" containerID="f55b81f300a3e1e865c5a9d80a4f844352ce6c88d0f4aad0c74f3a38442044c6"
Oct 30 05:04:07.995430 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.995407    4874 scope.go:122] "RemoveContainer" containerID="f55b81f300a3e1e865c5a9d80a4f844352ce6c88d0f4aad0c74f3a38442044c6"
Oct 30 05:04:07.995679 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:04:07.995661    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f55b81f300a3e1e865c5a9d80a4f844352ce6c88d0f4aad0c74f3a38442044c6\": container with ID starting with f55b81f300a3e1e865c5a9d80a4f844352ce6c88d0f4aad0c74f3a38442044c6 not found: ID does not exist" containerID="f55b81f300a3e1e865c5a9d80a4f844352ce6c88d0f4aad0c74f3a38442044c6"
Oct 30 05:04:07.995732 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:07.995686    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f55b81f300a3e1e865c5a9d80a4f844352ce6c88d0f4aad0c74f3a38442044c6"} err="failed to get container status \"f55b81f300a3e1e865c5a9d80a4f844352ce6c88d0f4aad0c74f3a38442044c6\": rpc error: code = NotFound desc = could not find container \"f55b81f300a3e1e865c5a9d80a4f844352ce6c88d0f4aad0c74f3a38442044c6\": container with ID starting with f55b81f300a3e1e865c5a9d80a4f844352ce6c88d0f4aad0c74f3a38442044c6 not found: ID does not exist"
Oct 30 05:04:08.034562 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:08.034534    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"]
Oct 30 05:04:08.045748 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:08.045726    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-fd27a9/test-model-fd27a9-kserve-559d8557d9-jqpq7"]
Oct 30 05:04:08.730974 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:08.730941    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26c033bd-b0b9-4b75-ab44-b0ccc7895677" path="/var/lib/kubelet/pods/26c033bd-b0b9-4b75-ab44-b0ccc7895677/volumes"
Oct 30 05:04:25.609230 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:04:25.609194    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:04:26.908674 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.908647    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"]
Oct 30 05:04:26.909344 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909324    4874 cpu_manager.go:414] "containerMap: removing container" podUID="90a79cd0-bb0f-40a4-bb6d-5099288a42e8" containerName="istio-proxy"
Oct 30 05:04:26.909344 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909341    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a79cd0-bb0f-40a4-bb6d-5099288a42e8" containerName="istio-proxy"
Oct 30 05:04:26.909499 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909352    4874 cpu_manager.go:414] "containerMap: removing container" podUID="06eecd27-740b-48d7-903c-182db0dae7ad" containerName="istio-proxy"
Oct 30 05:04:26.909499 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909360    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06eecd27-740b-48d7-903c-182db0dae7ad" containerName="istio-proxy"
Oct 30 05:04:26.909499 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909369    4874 cpu_manager.go:414] "containerMap: removing container" podUID="b35dc9fb-6f3c-4425-8360-6f52c94f1909" containerName="istio-proxy"
Oct 30 05:04:26.909499 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909377    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b35dc9fb-6f3c-4425-8360-6f52c94f1909" containerName="istio-proxy"
Oct 30 05:04:26.909499 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909392    4874 cpu_manager.go:414] "containerMap: removing container" podUID="b35dc9fb-6f3c-4425-8360-6f52c94f1909" containerName="istio-proxy"
Oct 30 05:04:26.909499 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909399    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b35dc9fb-6f3c-4425-8360-6f52c94f1909" containerName="istio-proxy"
Oct 30 05:04:26.909499 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909418    4874 cpu_manager.go:414] "containerMap: removing container" podUID="6ac7a2fc-ad35-41f6-babc-297ab9f7a97e" containerName="istio-proxy"
Oct 30 05:04:26.909499 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909423    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ac7a2fc-ad35-41f6-babc-297ab9f7a97e" containerName="istio-proxy"
Oct 30 05:04:26.909499 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909432    4874 cpu_manager.go:414] "containerMap: removing container" podUID="90a79cd0-bb0f-40a4-bb6d-5099288a42e8" containerName="istio-proxy"
Oct 30 05:04:26.909499 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909436    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a79cd0-bb0f-40a4-bb6d-5099288a42e8" containerName="istio-proxy"
Oct 30 05:04:26.909499 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909447    4874 cpu_manager.go:414] "containerMap: removing container" podUID="255f097a-7b21-4089-aec3-b0e3d964317e" containerName="cleanup"
Oct 30 05:04:26.909499 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909451    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="255f097a-7b21-4089-aec3-b0e3d964317e" containerName="cleanup"
Oct 30 05:04:26.909499 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909462    4874 cpu_manager.go:414] "containerMap: removing container" podUID="f33dee17-9ac8-4ea0-b547-94b24e63b4bb" containerName="istio-proxy"
Oct 30 05:04:26.909499 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909470    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f33dee17-9ac8-4ea0-b547-94b24e63b4bb" containerName="istio-proxy"
Oct 30 05:04:26.909499 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909481    4874 cpu_manager.go:414] "containerMap: removing container" podUID="6ac7a2fc-ad35-41f6-babc-297ab9f7a97e" containerName="istio-proxy"
Oct 30 05:04:26.909499 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909485    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ac7a2fc-ad35-41f6-babc-297ab9f7a97e" containerName="istio-proxy"
Oct 30 05:04:26.909499 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909493    4874 cpu_manager.go:414] "containerMap: removing container" podUID="26c033bd-b0b9-4b75-ab44-b0ccc7895677" containerName="main"
Oct 30 05:04:26.909499 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909498    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26c033bd-b0b9-4b75-ab44-b0ccc7895677" containerName="main"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909512    4874 cpu_manager.go:414] "containerMap: removing container" podUID="dd40b6ca-451d-4750-a621-920fa4d7425d" containerName="istio-proxy"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909517    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd40b6ca-451d-4750-a621-920fa4d7425d" containerName="istio-proxy"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909526    4874 cpu_manager.go:414] "containerMap: removing container" podUID="2b1ce86c-0464-4511-bed6-acefb0d3b841" containerName="maas-api"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909530    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b1ce86c-0464-4511-bed6-acefb0d3b841" containerName="maas-api"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909542    4874 cpu_manager.go:414] "containerMap: removing container" podUID="06eecd27-740b-48d7-903c-182db0dae7ad" containerName="istio-proxy"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909546    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06eecd27-740b-48d7-903c-182db0dae7ad" containerName="istio-proxy"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909553    4874 cpu_manager.go:414] "containerMap: removing container" podUID="cd62bd73-c45b-465d-b3e5-93b0bf16570a" containerName="maas-api"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909557    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd62bd73-c45b-465d-b3e5-93b0bf16570a" containerName="maas-api"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909567    4874 cpu_manager.go:414] "containerMap: removing container" podUID="f33dee17-9ac8-4ea0-b547-94b24e63b4bb" containerName="istio-proxy"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909572    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f33dee17-9ac8-4ea0-b547-94b24e63b4bb" containerName="istio-proxy"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909578    4874 cpu_manager.go:414] "containerMap: removing container" podUID="f1e88b87-85fa-4cce-ad5d-927bcddc23e9" containerName="main"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909582    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1e88b87-85fa-4cce-ad5d-927bcddc23e9" containerName="main"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909672    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b1ce86c-0464-4511-bed6-acefb0d3b841" containerName="maas-api"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909680    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="06eecd27-740b-48d7-903c-182db0dae7ad" containerName="istio-proxy"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909685    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd62bd73-c45b-465d-b3e5-93b0bf16570a" containerName="maas-api"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909691    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="f33dee17-9ac8-4ea0-b547-94b24e63b4bb" containerName="istio-proxy"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909695    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1e88b87-85fa-4cce-ad5d-927bcddc23e9" containerName="main"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909703    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a79cd0-bb0f-40a4-bb6d-5099288a42e8" containerName="istio-proxy"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909708    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="b35dc9fb-6f3c-4425-8360-6f52c94f1909" containerName="istio-proxy"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909715    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="b35dc9fb-6f3c-4425-8360-6f52c94f1909" containerName="istio-proxy"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909721    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ac7a2fc-ad35-41f6-babc-297ab9f7a97e" containerName="istio-proxy"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909728    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a79cd0-bb0f-40a4-bb6d-5099288a42e8" containerName="istio-proxy"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909733    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="255f097a-7b21-4089-aec3-b0e3d964317e" containerName="cleanup"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909739    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="f33dee17-9ac8-4ea0-b547-94b24e63b4bb" containerName="istio-proxy"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909761    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ac7a2fc-ad35-41f6-babc-297ab9f7a97e" containerName="istio-proxy"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909766    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="26c033bd-b0b9-4b75-ab44-b0ccc7895677" containerName="main"
Oct 30 05:04:26.910051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.909774    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd40b6ca-451d-4750-a621-920fa4d7425d" containerName="istio-proxy"
Oct 30 05:04:26.913402 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.913385    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:26.919806 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.919782    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"]
Oct 30 05:04:26.920554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.920535    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="06eecd27-740b-48d7-903c-182db0dae7ad" containerName="istio-proxy"
Oct 30 05:04:26.924028 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.924006    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:26.945472 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.938496    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"]
Oct 30 05:04:26.945472 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:26.944453    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"]
Oct 30 05:04:27.017351 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.017322    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7059d28d-3eca-44fa-80c4-ac634a59aaf7-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.017526 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.017367    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ea846a-5ce0-424a-bb80-ec759b592c37-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.017526 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.017392    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j8nd\" (UniqueName: \"kubernetes.io/projected/7059d28d-3eca-44fa-80c4-ac634a59aaf7-kube-api-access-4j8nd\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.017526 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.017508    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7059d28d-3eca-44fa-80c4-ac634a59aaf7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.017526 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.017527    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ea846a-5ce0-424a-bb80-ec759b592c37-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.017773 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.017574    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ea846a-5ce0-424a-bb80-ec759b592c37-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.017773 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.017631    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8tdg\" (UniqueName: \"kubernetes.io/projected/c2ea846a-5ce0-424a-bb80-ec759b592c37-kube-api-access-r8tdg\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.017773 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.017754    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7059d28d-3eca-44fa-80c4-ac634a59aaf7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.017919 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.017781    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7059d28d-3eca-44fa-80c4-ac634a59aaf7-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.017919 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.017804    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7059d28d-3eca-44fa-80c4-ac634a59aaf7-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.017919 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.017856    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ea846a-5ce0-424a-bb80-ec759b592c37-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.018073 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.017934    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ea846a-5ce0-424a-bb80-ec759b592c37-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.018073 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.017979    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7059d28d-3eca-44fa-80c4-ac634a59aaf7-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.018073 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.018046    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ea846a-5ce0-424a-bb80-ec759b592c37-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.018186 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.018106    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7059d28d-3eca-44fa-80c4-ac634a59aaf7-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.018186 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.018135    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7059d28d-3eca-44fa-80c4-ac634a59aaf7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.018287 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.018229    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ea846a-5ce0-424a-bb80-ec759b592c37-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.018341 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.018287    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ea846a-5ce0-424a-bb80-ec759b592c37-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.119972 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.119948    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7059d28d-3eca-44fa-80c4-ac634a59aaf7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.120155 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.119983    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7059d28d-3eca-44fa-80c4-ac634a59aaf7-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.120155 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.120005    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7059d28d-3eca-44fa-80c4-ac634a59aaf7-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.120155 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.120042    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ea846a-5ce0-424a-bb80-ec759b592c37-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.120155 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.120080    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ea846a-5ce0-424a-bb80-ec759b592c37-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.120155 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.120106    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7059d28d-3eca-44fa-80c4-ac634a59aaf7-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.120155 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.120131    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ea846a-5ce0-424a-bb80-ec759b592c37-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.120559 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.120181    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7059d28d-3eca-44fa-80c4-ac634a59aaf7-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.120559 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.120205    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7059d28d-3eca-44fa-80c4-ac634a59aaf7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.120559 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.120277    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ea846a-5ce0-424a-bb80-ec759b592c37-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.120559 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.120321    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ea846a-5ce0-424a-bb80-ec759b592c37-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.120559 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.120377    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7059d28d-3eca-44fa-80c4-ac634a59aaf7-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.120559 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.120424    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ea846a-5ce0-424a-bb80-ec759b592c37-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.120559 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.120454    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j8nd\" (UniqueName: \"kubernetes.io/projected/7059d28d-3eca-44fa-80c4-ac634a59aaf7-kube-api-access-4j8nd\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.120559 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.120489    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7059d28d-3eca-44fa-80c4-ac634a59aaf7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.120559 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.120491    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ea846a-5ce0-424a-bb80-ec759b592c37-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.120559 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.120500    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7059d28d-3eca-44fa-80c4-ac634a59aaf7-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.120559 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.120507    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7059d28d-3eca-44fa-80c4-ac634a59aaf7-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.120559 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.120517    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ea846a-5ce0-424a-bb80-ec759b592c37-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.121361 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.120605    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7059d28d-3eca-44fa-80c4-ac634a59aaf7-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.121361 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.120611    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ea846a-5ce0-424a-bb80-ec759b592c37-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.121361 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.120660    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8tdg\" (UniqueName: \"kubernetes.io/projected/c2ea846a-5ce0-424a-bb80-ec759b592c37-kube-api-access-r8tdg\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.121361 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.120765    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ea846a-5ce0-424a-bb80-ec759b592c37-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.121361 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.120790    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7059d28d-3eca-44fa-80c4-ac634a59aaf7-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.121361 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.120843    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ea846a-5ce0-424a-bb80-ec759b592c37-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.121361 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.121180    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ea846a-5ce0-424a-bb80-ec759b592c37-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.121704 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.121441    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ea846a-5ce0-424a-bb80-ec759b592c37-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.121704 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.121456    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7059d28d-3eca-44fa-80c4-ac634a59aaf7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.123348 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.123315    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ea846a-5ce0-424a-bb80-ec759b592c37-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.123497 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.123477    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7059d28d-3eca-44fa-80c4-ac634a59aaf7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.123612 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.123591    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7059d28d-3eca-44fa-80c4-ac634a59aaf7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.123820 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.123800    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ea846a-5ce0-424a-bb80-ec759b592c37-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.142898 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.142875    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ea846a-5ce0-424a-bb80-ec759b592c37-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.143212 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.143189    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8tdg\" (UniqueName: \"kubernetes.io/projected/c2ea846a-5ce0-424a-bb80-ec759b592c37-kube-api-access-r8tdg\") pod \"maas-default-gateway-openshift-default-7f694df8bc-x8mpd\" (UID: \"c2ea846a-5ce0-424a-bb80-ec759b592c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.143633 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.143610    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7059d28d-3eca-44fa-80c4-ac634a59aaf7-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.143753 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.143737    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j8nd\" (UniqueName: \"kubernetes.io/projected/7059d28d-3eca-44fa-80c4-ac634a59aaf7-kube-api-access-4j8nd\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fh868\" (UID: \"7059d28d-3eca-44fa-80c4-ac634a59aaf7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.230391 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.230368    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:27.244495 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.244473    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:27.571483 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.571451    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"]
Oct 30 05:04:27.572861 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:04:27.572834    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7059d28d_3eca_44fa_80c4_ac634a59aaf7.slice/crio-ad96c243d86b26e62498e344515111f74c1e5fca17d1a69d7c5cb2e582161a27 WatchSource:0}: Error finding container ad96c243d86b26e62498e344515111f74c1e5fca17d1a69d7c5cb2e582161a27: Status 404 returned error can't find the container with id ad96c243d86b26e62498e344515111f74c1e5fca17d1a69d7c5cb2e582161a27
Oct 30 05:04:27.574949 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.574929    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"]
Oct 30 05:04:27.575698 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.575668    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 05:04:27.575750 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.575730    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 05:04:27.575790 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.575760    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 05:04:27.586434 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:04:27.586409    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ea846a_5ce0_424a_bb80_ec759b592c37.slice/crio-750db2d61c557f6f2a8341b81490edc31ae31076ec2f6dfbdd71cd242c6aa80f WatchSource:0}: Error finding container 750db2d61c557f6f2a8341b81490edc31ae31076ec2f6dfbdd71cd242c6aa80f: Status 404 returned error can't find the container with id 750db2d61c557f6f2a8341b81490edc31ae31076ec2f6dfbdd71cd242c6aa80f
Oct 30 05:04:27.589293 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.589262    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 05:04:27.589397 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.589337    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 05:04:27.589397 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:27.589369    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 05:04:28.134844 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:28.134693    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd" event={"ID":"c2ea846a-5ce0-424a-bb80-ec759b592c37","Type":"ContainerStarted","Data":"72dca9e78249fe547d1d3da3df4eef2a407385c7055ae01dd555193c571fce8a"}
Oct 30 05:04:28.134844 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:28.134803    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd" event={"ID":"c2ea846a-5ce0-424a-bb80-ec759b592c37","Type":"ContainerStarted","Data":"750db2d61c557f6f2a8341b81490edc31ae31076ec2f6dfbdd71cd242c6aa80f"}
Oct 30 05:04:28.136845 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:28.136815    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868" event={"ID":"7059d28d-3eca-44fa-80c4-ac634a59aaf7","Type":"ContainerStarted","Data":"0d3c83680edd62a39bf2ea6e6a4f41eca279b3253cb705e6bd24630aa3c9a906"}
Oct 30 05:04:28.136998 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:28.136866    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868" event={"ID":"7059d28d-3eca-44fa-80c4-ac634a59aaf7","Type":"ContainerStarted","Data":"ad96c243d86b26e62498e344515111f74c1e5fca17d1a69d7c5cb2e582161a27"}
Oct 30 05:04:28.179948 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:28.179884    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd" podStartSLOduration=2.179866785 podStartE2EDuration="2.179866785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:04:28.177224812 +0000 UTC m=+3987.589620402" watchObservedRunningTime="2026-10-30 05:04:28.179866785 +0000 UTC m=+3987.592262359"
Oct 30 05:04:28.222890 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:28.222845    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868" podStartSLOduration=2.222832249 podStartE2EDuration="2.222832249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:04:28.219383624 +0000 UTC m=+3987.631779196" watchObservedRunningTime="2026-10-30 05:04:28.222832249 +0000 UTC m=+3987.635227822"
Oct 30 05:04:28.231085 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:28.231059    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:28.232309 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:28.232291    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fh868 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 30 05:04:28.232364 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:28.232330    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868" podUID="7059d28d-3eca-44fa-80c4-ac634a59aaf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 30 05:04:28.244671 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:28.244643    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:28.246004 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:28.245982    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-x8mpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 05:04:28.246114 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:28.246023    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd" podUID="c2ea846a-5ce0-424a-bb80-ec759b592c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 05:04:29.231364 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:29.231329    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fh868 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 30 05:04:29.231804 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:29.231392    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868" podUID="7059d28d-3eca-44fa-80c4-ac634a59aaf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 30 05:04:29.244887 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:29.244860    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-x8mpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 05:04:29.244967 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:29.244919    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd" podUID="c2ea846a-5ce0-424a-bb80-ec759b592c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 05:04:30.231342 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:30.231308    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fh868 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 30 05:04:30.231531 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:30.231356    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868" podUID="7059d28d-3eca-44fa-80c4-ac634a59aaf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 30 05:04:30.244761 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:30.244734    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-x8mpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 05:04:30.244842 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:30.244785    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd" podUID="c2ea846a-5ce0-424a-bb80-ec759b592c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 05:04:31.230940 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:31.230908    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fh868 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 30 05:04:31.231149 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:31.230964    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868" podUID="7059d28d-3eca-44fa-80c4-ac634a59aaf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 30 05:04:31.245172 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:31.245143    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-x8mpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 05:04:31.245643 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:31.245620    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd" podUID="c2ea846a-5ce0-424a-bb80-ec759b592c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 05:04:32.231427 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:32.231389    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fh868 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 30 05:04:32.231644 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:32.231446    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868" podUID="7059d28d-3eca-44fa-80c4-ac634a59aaf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 30 05:04:32.245723 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:32.245695    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-x8mpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 05:04:32.246056 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:32.245738    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd" podUID="c2ea846a-5ce0-424a-bb80-ec759b592c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 05:04:32.326869 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:32.326828    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:04:33.231586 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:33.231546    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fh868 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 30 05:04:33.231868 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:33.231596    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868" podUID="7059d28d-3eca-44fa-80c4-ac634a59aaf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 30 05:04:33.245585 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:33.245558    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-x8mpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 05:04:33.245722 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:33.245599    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd" podUID="c2ea846a-5ce0-424a-bb80-ec759b592c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 05:04:34.231525 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:34.231491    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fh868 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 30 05:04:34.231917 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:34.231554    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868" podUID="7059d28d-3eca-44fa-80c4-ac634a59aaf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 30 05:04:34.245352 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:34.245326    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-x8mpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 05:04:34.245468 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:34.245369    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd" podUID="c2ea846a-5ce0-424a-bb80-ec759b592c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 05:04:35.230992 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:35.230956    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fh868 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 30 05:04:35.231170 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:35.231020    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868" podUID="7059d28d-3eca-44fa-80c4-ac634a59aaf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 30 05:04:35.245759 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:35.245727    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-x8mpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 05:04:35.246110 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:35.245784    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd" podUID="c2ea846a-5ce0-424a-bb80-ec759b592c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 05:04:36.734747 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:36.734715    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:36.736118 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:36.736095    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:36.739194 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:36.739153    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd"
Oct 30 05:04:36.739624 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:36.739605    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:36.740141 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:36.740121    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:36.740881 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:04:36.740865    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868"
Oct 30 05:04:55.610646 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:04:55.610607    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:05:24.628395 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:05:24.628360    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:05:25.615881 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:05:25.615841    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:05:55.676354 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:05:55.676313    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:06:04.849618 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:04.849579    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"]
Oct 30 05:06:16.563599 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:16.563517    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:06:19.891885 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:19.891843    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r" podUID="761c729c-5767-4b78-b916-0841ad83d9e7" containerName="main" containerID="cri-o://783770281ff2b7ae351180a19f36274df88ba4751f36c43b72c02b6bfdcd7eaf" gracePeriod=45
Oct 30 05:06:20.060808 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:20.060771    4874 generic.go:356] "Generic (PLEG): container finished" podID="761c729c-5767-4b78-b916-0841ad83d9e7" containerID="783770281ff2b7ae351180a19f36274df88ba4751f36c43b72c02b6bfdcd7eaf" exitCode=0
Oct 30 05:06:20.061062 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:20.060814    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r" event={"ID":"761c729c-5767-4b78-b916-0841ad83d9e7","Type":"ContainerDied","Data":"783770281ff2b7ae351180a19f36274df88ba4751f36c43b72c02b6bfdcd7eaf"}
Oct 30 05:06:20.354154 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:20.354129    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:06:20.469516 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:20.469443    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/761c729c-5767-4b78-b916-0841ad83d9e7-tls-certs\" (UniqueName: \"kubernetes.io/secret/761c729c-5767-4b78-b916-0841ad83d9e7-tls-certs\") pod \"761c729c-5767-4b78-b916-0841ad83d9e7\" (UID: \"761c729c-5767-4b78-b916-0841ad83d9e7\") "
Oct 30 05:06:20.469516 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:20.469495    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/761c729c-5767-4b78-b916-0841ad83d9e7-kube-api-access-8xft6\" (UniqueName: \"kubernetes.io/projected/761c729c-5767-4b78-b916-0841ad83d9e7-kube-api-access-8xft6\") pod \"761c729c-5767-4b78-b916-0841ad83d9e7\" (UID: \"761c729c-5767-4b78-b916-0841ad83d9e7\") "
Oct 30 05:06:20.469731 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:20.469580    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-home\" (UniqueName: \"kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-home\") pod \"761c729c-5767-4b78-b916-0841ad83d9e7\" (UID: \"761c729c-5767-4b78-b916-0841ad83d9e7\") "
Oct 30 05:06:20.469731 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:20.469696    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-dshm\") pod \"761c729c-5767-4b78-b916-0841ad83d9e7\" (UID: \"761c729c-5767-4b78-b916-0841ad83d9e7\") "
Oct 30 05:06:20.469846 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:20.469805    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-home" pod "761c729c-5767-4b78-b916-0841ad83d9e7" (UID: "761c729c-5767-4b78-b916-0841ad83d9e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:06:20.469904 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:20.469843    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-model-cache\") pod \"761c729c-5767-4b78-b916-0841ad83d9e7\" (UID: \"761c729c-5767-4b78-b916-0841ad83d9e7\") "
Oct 30 05:06:20.469957 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:20.469911    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-tmp-dir\") pod \"761c729c-5767-4b78-b916-0841ad83d9e7\" (UID: \"761c729c-5767-4b78-b916-0841ad83d9e7\") "
Oct 30 05:06:20.470111 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:20.470084    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-model-cache" pod "761c729c-5767-4b78-b916-0841ad83d9e7" (UID: "761c729c-5767-4b78-b916-0841ad83d9e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:06:20.470232 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:20.470205    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-tmp-dir" pod "761c729c-5767-4b78-b916-0841ad83d9e7" (UID: "761c729c-5767-4b78-b916-0841ad83d9e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:06:20.470642 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:20.470618    4874 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-tmp-dir\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:06:20.470681 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:20.470649    4874 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-home\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:06:20.470681 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:20.470664    4874 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-model-cache\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:06:20.471771 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:20.471752    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/761c729c-5767-4b78-b916-0841ad83d9e7-tls-certs" pod "761c729c-5767-4b78-b916-0841ad83d9e7" (UID: "761c729c-5767-4b78-b916-0841ad83d9e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:06:20.471889 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:20.471810    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/761c729c-5767-4b78-b916-0841ad83d9e7-kube-api-access-8xft6" pod "761c729c-5767-4b78-b916-0841ad83d9e7" (UID: "761c729c-5767-4b78-b916-0841ad83d9e7"). InnerVolumeSpecName "kube-api-access-8xft6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:06:20.471889 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:20.471840    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-dshm" pod "761c729c-5767-4b78-b916-0841ad83d9e7" (UID: "761c729c-5767-4b78-b916-0841ad83d9e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:06:20.572135 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:20.572108    4874 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/761c729c-5767-4b78-b916-0841ad83d9e7-tls-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:06:20.572135 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:20.572132    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xft6\" (UniqueName: \"kubernetes.io/projected/761c729c-5767-4b78-b916-0841ad83d9e7-kube-api-access-8xft6\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:06:20.572341 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:20.572143    4874 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/761c729c-5767-4b78-b916-0841ad83d9e7-dshm\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:06:21.070106 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:21.070025    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"
Oct 30 05:06:21.070106 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:21.070046    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r" event={"ID":"761c729c-5767-4b78-b916-0841ad83d9e7","Type":"ContainerDied","Data":"f26a680e106c062331e1bc294cf31d1de25245898f89e5ebdc6bb5bb5a319557"}
Oct 30 05:06:21.070106 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:21.070103    4874 scope.go:122] "RemoveContainer" containerID="783770281ff2b7ae351180a19f36274df88ba4751f36c43b72c02b6bfdcd7eaf"
Oct 30 05:06:21.120353 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:21.120320    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"]
Oct 30 05:06:21.124899 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:21.124875    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9f158a/test-model-9f158a-kserve-66f8cd44dc-jxz9r"]
Oct 30 05:06:22.729897 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:22.729864    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="761c729c-5767-4b78-b916-0841ad83d9e7" path="/var/lib/kubelet/pods/761c729c-5767-4b78-b916-0841ad83d9e7/volumes"
Oct 30 05:06:24.943407 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:24.943055    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp"]
Oct 30 05:06:24.943407 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:24.943342    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp" podUID="318e0d5c-3645-4477-8a63-13849cca77fc" containerName="maas-api" containerID="cri-o://6a3a8c5e7c01891c7cc4c1d38dce0f297a229571b5784d705c02afc180b98a5d" gracePeriod=30
Oct 30 05:06:25.099080 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:25.099047    4874 generic.go:356] "Generic (PLEG): container finished" podID="318e0d5c-3645-4477-8a63-13849cca77fc" containerID="6a3a8c5e7c01891c7cc4c1d38dce0f297a229571b5784d705c02afc180b98a5d" exitCode=0
Oct 30 05:06:25.099263 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:25.099119    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp" event={"ID":"318e0d5c-3645-4477-8a63-13849cca77fc","Type":"ContainerDied","Data":"6a3a8c5e7c01891c7cc4c1d38dce0f297a229571b5784d705c02afc180b98a5d"}
Oct 30 05:06:25.262861 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:25.259265    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp"
Oct 30 05:06:25.329720 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:25.329691    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/318e0d5c-3645-4477-8a63-13849cca77fc-kube-api-access-m7lq8\" (UniqueName: \"kubernetes.io/projected/318e0d5c-3645-4477-8a63-13849cca77fc-kube-api-access-m7lq8\") pod \"318e0d5c-3645-4477-8a63-13849cca77fc\" (UID: \"318e0d5c-3645-4477-8a63-13849cca77fc\") "
Oct 30 05:06:25.329917 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:25.329760    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/318e0d5c-3645-4477-8a63-13849cca77fc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/318e0d5c-3645-4477-8a63-13849cca77fc-maas-api-tls\") pod \"318e0d5c-3645-4477-8a63-13849cca77fc\" (UID: \"318e0d5c-3645-4477-8a63-13849cca77fc\") "
Oct 30 05:06:25.329992 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:25.329945    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/318e0d5c-3645-4477-8a63-13849cca77fc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/318e0d5c-3645-4477-8a63-13849cca77fc-openshift-service-ca\") pod \"318e0d5c-3645-4477-8a63-13849cca77fc\" (UID: \"318e0d5c-3645-4477-8a63-13849cca77fc\") "
Oct 30 05:06:25.330051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:25.329999    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/318e0d5c-3645-4477-8a63-13849cca77fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/318e0d5c-3645-4477-8a63-13849cca77fc-metrics-certs\") pod \"318e0d5c-3645-4477-8a63-13849cca77fc\" (UID: \"318e0d5c-3645-4477-8a63-13849cca77fc\") "
Oct 30 05:06:25.331635 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:25.330331    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/318e0d5c-3645-4477-8a63-13849cca77fc-openshift-service-ca" pod "318e0d5c-3645-4477-8a63-13849cca77fc" (UID: "318e0d5c-3645-4477-8a63-13849cca77fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:06:25.331635 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:25.330793    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/318e0d5c-3645-4477-8a63-13849cca77fc-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:06:25.332897 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:25.332870    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/318e0d5c-3645-4477-8a63-13849cca77fc-kube-api-access-m7lq8" pod "318e0d5c-3645-4477-8a63-13849cca77fc" (UID: "318e0d5c-3645-4477-8a63-13849cca77fc"). InnerVolumeSpecName "kube-api-access-m7lq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:06:25.334519 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:25.334479    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/318e0d5c-3645-4477-8a63-13849cca77fc-metrics-certs" pod "318e0d5c-3645-4477-8a63-13849cca77fc" (UID: "318e0d5c-3645-4477-8a63-13849cca77fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:06:25.343277 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:25.336358    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/318e0d5c-3645-4477-8a63-13849cca77fc-maas-api-tls" pod "318e0d5c-3645-4477-8a63-13849cca77fc" (UID: "318e0d5c-3645-4477-8a63-13849cca77fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:06:25.493370 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:25.479368    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/318e0d5c-3645-4477-8a63-13849cca77fc-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:06:25.493370 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:25.479399    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7lq8\" (UniqueName: \"kubernetes.io/projected/318e0d5c-3645-4477-8a63-13849cca77fc-kube-api-access-m7lq8\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:06:25.493370 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:25.479411    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/318e0d5c-3645-4477-8a63-13849cca77fc-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:06:25.609875 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:06:25.609842    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:06:26.106659 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:26.106626    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp"
Oct 30 05:06:26.107139 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:26.106678    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp" event={"ID":"318e0d5c-3645-4477-8a63-13849cca77fc","Type":"ContainerDied","Data":"acf164ee0ff1722e03084f03cf4d3c5360300027d0cf6cb331e0379702c0159a"}
Oct 30 05:06:26.107139 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:26.106745    4874 scope.go:122] "RemoveContainer" containerID="6a3a8c5e7c01891c7cc4c1d38dce0f297a229571b5784d705c02afc180b98a5d"
Oct 30 05:06:26.156540 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:26.156513    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp"]
Oct 30 05:06:26.167648 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:26.167626    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7700a7-5f4678c764-22ddp"]
Oct 30 05:06:26.739861 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:06:26.739827    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="318e0d5c-3645-4477-8a63-13849cca77fc" path="/var/lib/kubelet/pods/318e0d5c-3645-4477-8a63-13849cca77fc/volumes"
Oct 30 05:06:55.609353 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:06:55.609314    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:07:00.258462 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:00.258424    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:07:01.394807 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:01.394769    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw"]
Oct 30 05:07:01.395370 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:01.395064    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw" podUID="e51a82fb-9ec5-4fd4-92c7-6ae38029739b" containerName="maas-api" containerID="cri-o://a058f4ec21f99714d09c624336ac3d028768a0341e759b363f8ec9aa2511ff40" gracePeriod=30
Oct 30 05:07:01.747306 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:01.745087    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw"
Oct 30 05:07:01.885276 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:01.868499    4874 scope.go:122] "RemoveContainer" containerID="a058f4ec21f99714d09c624336ac3d028768a0341e759b363f8ec9aa2511ff40"
Oct 30 05:07:01.907117 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:01.907091    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-metrics-certs\") pod \"e51a82fb-9ec5-4fd4-92c7-6ae38029739b\" (UID: \"e51a82fb-9ec5-4fd4-92c7-6ae38029739b\") "
Oct 30 05:07:01.907496 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:01.907481    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-openshift-service-ca\") pod \"e51a82fb-9ec5-4fd4-92c7-6ae38029739b\" (UID: \"e51a82fb-9ec5-4fd4-92c7-6ae38029739b\") "
Oct 30 05:07:01.907705 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:01.907694    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-maas-api-tls\") pod \"e51a82fb-9ec5-4fd4-92c7-6ae38029739b\" (UID: \"e51a82fb-9ec5-4fd4-92c7-6ae38029739b\") "
Oct 30 05:07:01.907910 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:01.907897    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-kube-api-access-n6pw9\" (UniqueName: \"kubernetes.io/projected/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-kube-api-access-n6pw9\") pod \"e51a82fb-9ec5-4fd4-92c7-6ae38029739b\" (UID: \"e51a82fb-9ec5-4fd4-92c7-6ae38029739b\") "
Oct 30 05:07:01.912990 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:01.908407    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-openshift-service-ca" pod "e51a82fb-9ec5-4fd4-92c7-6ae38029739b" (UID: "e51a82fb-9ec5-4fd4-92c7-6ae38029739b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:07:01.926878 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:01.926845    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-kube-api-access-n6pw9" pod "e51a82fb-9ec5-4fd4-92c7-6ae38029739b" (UID: "e51a82fb-9ec5-4fd4-92c7-6ae38029739b"). InnerVolumeSpecName "kube-api-access-n6pw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:07:01.957158 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:01.957116    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-maas-api-tls" pod "e51a82fb-9ec5-4fd4-92c7-6ae38029739b" (UID: "e51a82fb-9ec5-4fd4-92c7-6ae38029739b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:07:01.962692 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:01.962660    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-metrics-certs" pod "e51a82fb-9ec5-4fd4-92c7-6ae38029739b" (UID: "e51a82fb-9ec5-4fd4-92c7-6ae38029739b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:07:02.013604 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:02.013578    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6pw9\" (UniqueName: \"kubernetes.io/projected/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-kube-api-access-n6pw9\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:02.013604 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:02.013604    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:02.013788 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:02.013613    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:02.013788 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:02.013621    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e51a82fb-9ec5-4fd4-92c7-6ae38029739b-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:02.378939 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:02.378849    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw" event={"ID":"e51a82fb-9ec5-4fd4-92c7-6ae38029739b","Type":"ContainerDied","Data":"a058f4ec21f99714d09c624336ac3d028768a0341e759b363f8ec9aa2511ff40"}
Oct 30 05:07:02.378939 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:02.378918    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw" event={"ID":"e51a82fb-9ec5-4fd4-92c7-6ae38029739b","Type":"ContainerDied","Data":"712a01464cd08e2be4b5cfdeac4680d3fdee04fede550d421aa8384f98770380"}
Oct 30 05:07:02.379156 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:02.378946    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw"
Oct 30 05:07:02.455801 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:02.455768    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw"]
Oct 30 05:07:02.461556 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:02.461532    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9f158a-767d7f9844-z8kkw"]
Oct 30 05:07:02.731451 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:02.731420    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e51a82fb-9ec5-4fd4-92c7-6ae38029739b" path="/var/lib/kubelet/pods/e51a82fb-9ec5-4fd4-92c7-6ae38029739b/volumes"
Oct 30 05:07:17.182805 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:17.182770    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb"]
Oct 30 05:07:17.183264 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:17.183015    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb" podUID="a9735d04-5cad-4502-8f8b-521b47d5c037" containerName="maas-api" containerID="cri-o://c454f80f25b215c91220f8b1d74ae2ed45c8e54b8dc55a77ac325e42ce38b770" gracePeriod=30
Oct 30 05:07:17.304368 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:17.304333    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb" podUID="a9735d04-5cad-4502-8f8b-521b47d5c037" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.172:8443/health\": dial tcp 10.134.0.172:8443: connect: connection refused"
Oct 30 05:07:17.518626 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:17.518588    4874 generic.go:356] "Generic (PLEG): container finished" podID="a9735d04-5cad-4502-8f8b-521b47d5c037" containerID="c454f80f25b215c91220f8b1d74ae2ed45c8e54b8dc55a77ac325e42ce38b770" exitCode=0
Oct 30 05:07:17.518806 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:17.518787    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb" event={"ID":"a9735d04-5cad-4502-8f8b-521b47d5c037","Type":"ContainerDied","Data":"c454f80f25b215c91220f8b1d74ae2ed45c8e54b8dc55a77ac325e42ce38b770"}
Oct 30 05:07:17.548357 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:17.548323    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb"
Oct 30 05:07:17.696267 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:17.696111    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9735d04-5cad-4502-8f8b-521b47d5c037-kube-api-access-sfk47\" (UniqueName: \"kubernetes.io/projected/a9735d04-5cad-4502-8f8b-521b47d5c037-kube-api-access-sfk47\") pod \"a9735d04-5cad-4502-8f8b-521b47d5c037\" (UID: \"a9735d04-5cad-4502-8f8b-521b47d5c037\") "
Oct 30 05:07:17.696267 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:17.696183    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9735d04-5cad-4502-8f8b-521b47d5c037-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9735d04-5cad-4502-8f8b-521b47d5c037-openshift-service-ca\") pod \"a9735d04-5cad-4502-8f8b-521b47d5c037\" (UID: \"a9735d04-5cad-4502-8f8b-521b47d5c037\") "
Oct 30 05:07:17.696536 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:17.696328    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9735d04-5cad-4502-8f8b-521b47d5c037-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9735d04-5cad-4502-8f8b-521b47d5c037-maas-api-tls\") pod \"a9735d04-5cad-4502-8f8b-521b47d5c037\" (UID: \"a9735d04-5cad-4502-8f8b-521b47d5c037\") "
Oct 30 05:07:17.696536 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:17.696406    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9735d04-5cad-4502-8f8b-521b47d5c037-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9735d04-5cad-4502-8f8b-521b47d5c037-metrics-certs\") pod \"a9735d04-5cad-4502-8f8b-521b47d5c037\" (UID: \"a9735d04-5cad-4502-8f8b-521b47d5c037\") "
Oct 30 05:07:17.734258 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:17.724291    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9735d04-5cad-4502-8f8b-521b47d5c037-metrics-certs" pod "a9735d04-5cad-4502-8f8b-521b47d5c037" (UID: "a9735d04-5cad-4502-8f8b-521b47d5c037"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:07:17.734258 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:17.727504    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9735d04-5cad-4502-8f8b-521b47d5c037-kube-api-access-sfk47" pod "a9735d04-5cad-4502-8f8b-521b47d5c037" (UID: "a9735d04-5cad-4502-8f8b-521b47d5c037"). InnerVolumeSpecName "kube-api-access-sfk47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:07:17.734258 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:17.727826    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9735d04-5cad-4502-8f8b-521b47d5c037-openshift-service-ca" pod "a9735d04-5cad-4502-8f8b-521b47d5c037" (UID: "a9735d04-5cad-4502-8f8b-521b47d5c037"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:07:17.734258 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:17.730675    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9735d04-5cad-4502-8f8b-521b47d5c037-maas-api-tls" pod "a9735d04-5cad-4502-8f8b-521b47d5c037" (UID: "a9735d04-5cad-4502-8f8b-521b47d5c037"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:07:17.798260 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:17.798157    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9735d04-5cad-4502-8f8b-521b47d5c037-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:17.798260 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:17.798190    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9735d04-5cad-4502-8f8b-521b47d5c037-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:17.798260 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:17.798204    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfk47\" (UniqueName: \"kubernetes.io/projected/a9735d04-5cad-4502-8f8b-521b47d5c037-kube-api-access-sfk47\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:17.798260 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:17.798218    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9735d04-5cad-4502-8f8b-521b47d5c037-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:18.526596 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:18.526567    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb"
Oct 30 05:07:18.526596 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:18.526572    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb" event={"ID":"a9735d04-5cad-4502-8f8b-521b47d5c037","Type":"ContainerDied","Data":"5fd893425b1661475c6f59ad15ef690e1d6f757b12d7d2335e337a9ca62984d8"}
Oct 30 05:07:18.527122 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:18.526639    4874 scope.go:122] "RemoveContainer" containerID="c454f80f25b215c91220f8b1d74ae2ed45c8e54b8dc55a77ac325e42ce38b770"
Oct 30 05:07:18.580318 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:18.580284    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb"]
Oct 30 05:07:18.589037 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:18.589008    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e592d7-d7fcffc65-lmhsb"]
Oct 30 05:07:18.730432 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:18.730402    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9735d04-5cad-4502-8f8b-521b47d5c037" path="/var/lib/kubelet/pods/a9735d04-5cad-4502-8f8b-521b47d5c037/volumes"
Oct 30 05:07:22.508641 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:22.508603    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"]
Oct 30 05:07:22.509051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:22.508859    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6" podUID="8ca71311-3f21-47db-98a2-a9f95fbdebe9" containerName="istio-proxy" containerID="cri-o://5b75178ca380fc44957225360994c9d224faee93c62c15d98c74750c3345911e" gracePeriod=30
Oct 30 05:07:25.611119 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:07:25.611081    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:07:27.590583 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.590552    4874 patch_prober.go:29] interesting pod/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 05:07:27.590958 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.590604    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6" podUID="8ca71311-3f21-47db-98a2-a9f95fbdebe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 05:07:27.774272 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.774192    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 05:07:27.859146 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.859115    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-workload-certs\") pod \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") "
Oct 30 05:07:27.859341 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.859202    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-credential-socket\") pod \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") "
Oct 30 05:07:27.859341 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.859328    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-token\" (UniqueName: \"kubernetes.io/projected/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-token\") pod \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") "
Oct 30 05:07:27.859481 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.859361    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-data\") pod \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") "
Oct 30 05:07:27.859481 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.859412    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-podinfo\") pod \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") "
Oct 30 05:07:27.859481 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.859409    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-workload-certs" pod "8ca71311-3f21-47db-98a2-a9f95fbdebe9" (UID: "8ca71311-3f21-47db-98a2-a9f95fbdebe9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:07:27.859481 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.859439    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ca71311-3f21-47db-98a2-a9f95fbdebe9-kube-api-access-nzwn5\" (UniqueName: \"kubernetes.io/projected/8ca71311-3f21-47db-98a2-a9f95fbdebe9-kube-api-access-nzwn5\") pod \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") "
Oct 30 05:07:27.859481 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.859468    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-workload-socket\") pod \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") "
Oct 30 05:07:27.859733 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.859513    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-envoy\") pod \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") "
Oct 30 05:07:27.859733 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.859515    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-credential-socket" pod "8ca71311-3f21-47db-98a2-a9f95fbdebe9" (UID: "8ca71311-3f21-47db-98a2-a9f95fbdebe9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:07:27.859733 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.859543    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istiod-ca-cert\") pod \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\" (UID: \"8ca71311-3f21-47db-98a2-a9f95fbdebe9\") "
Oct 30 05:07:27.859733 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.859631    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-data" pod "8ca71311-3f21-47db-98a2-a9f95fbdebe9" (UID: "8ca71311-3f21-47db-98a2-a9f95fbdebe9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:07:27.860596 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.860197    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:27.860596 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.860223    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:27.860596 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.860266    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:27.860596 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.860337    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-workload-socket" pod "8ca71311-3f21-47db-98a2-a9f95fbdebe9" (UID: "8ca71311-3f21-47db-98a2-a9f95fbdebe9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:07:27.860596 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.860564    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istiod-ca-cert" pod "8ca71311-3f21-47db-98a2-a9f95fbdebe9" (UID: "8ca71311-3f21-47db-98a2-a9f95fbdebe9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:07:27.862394 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.862199    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-token" pod "8ca71311-3f21-47db-98a2-a9f95fbdebe9" (UID: "8ca71311-3f21-47db-98a2-a9f95fbdebe9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:07:27.862394 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.862311    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca71311-3f21-47db-98a2-a9f95fbdebe9-kube-api-access-nzwn5" pod "8ca71311-3f21-47db-98a2-a9f95fbdebe9" (UID: "8ca71311-3f21-47db-98a2-a9f95fbdebe9"). InnerVolumeSpecName "kube-api-access-nzwn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:07:27.862568 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.862411    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-podinfo" pod "8ca71311-3f21-47db-98a2-a9f95fbdebe9" (UID: "8ca71311-3f21-47db-98a2-a9f95fbdebe9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:07:27.862568 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.862437    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-envoy" pod "8ca71311-3f21-47db-98a2-a9f95fbdebe9" (UID: "8ca71311-3f21-47db-98a2-a9f95fbdebe9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:07:27.961375 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.961343    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:27.961375 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.961371    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzwn5\" (UniqueName: \"kubernetes.io/projected/8ca71311-3f21-47db-98a2-a9f95fbdebe9-kube-api-access-nzwn5\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:27.961569 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.961386    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:27.961569 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.961399    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:27.961569 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.961411    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:27.961569 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:27.961421    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ca71311-3f21-47db-98a2-a9f95fbdebe9-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:28.593069 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:28.593033    4874 generic.go:356] "Generic (PLEG): container finished" podID="8ca71311-3f21-47db-98a2-a9f95fbdebe9" containerID="5b75178ca380fc44957225360994c9d224faee93c62c15d98c74750c3345911e" exitCode=0
Oct 30 05:07:28.593571 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:28.593091    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"
Oct 30 05:07:28.593571 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:28.593124    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6" event={"ID":"8ca71311-3f21-47db-98a2-a9f95fbdebe9","Type":"ContainerDied","Data":"5b75178ca380fc44957225360994c9d224faee93c62c15d98c74750c3345911e"}
Oct 30 05:07:28.593571 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:28.593178    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6" event={"ID":"8ca71311-3f21-47db-98a2-a9f95fbdebe9","Type":"ContainerDied","Data":"fc5a94729b9b3f7166f70b71f783d07a16122ee0860c615f36512deb745ff924"}
Oct 30 05:07:28.593571 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:28.593199    4874 scope.go:122] "RemoveContainer" containerID="5b75178ca380fc44957225360994c9d224faee93c62c15d98c74750c3345911e"
Oct 30 05:07:28.604524 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:28.604440    4874 scope.go:122] "RemoveContainer" containerID="0c21bad86504624583c3386731842623b64602c0a4b1e7a3d7331545021f642b"
Oct 30 05:07:28.668833 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:28.668655    4874 scope.go:122] "RemoveContainer" containerID="5b75178ca380fc44957225360994c9d224faee93c62c15d98c74750c3345911e"
Oct 30 05:07:28.669783 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:07:28.669751    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b75178ca380fc44957225360994c9d224faee93c62c15d98c74750c3345911e\": container with ID starting with 5b75178ca380fc44957225360994c9d224faee93c62c15d98c74750c3345911e not found: ID does not exist" containerID="5b75178ca380fc44957225360994c9d224faee93c62c15d98c74750c3345911e"
Oct 30 05:07:28.669962 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:28.669930    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b75178ca380fc44957225360994c9d224faee93c62c15d98c74750c3345911e"} err="failed to get container status \"5b75178ca380fc44957225360994c9d224faee93c62c15d98c74750c3345911e\": rpc error: code = NotFound desc = could not find container \"5b75178ca380fc44957225360994c9d224faee93c62c15d98c74750c3345911e\": container with ID starting with 5b75178ca380fc44957225360994c9d224faee93c62c15d98c74750c3345911e not found: ID does not exist"
Oct 30 05:07:28.669962 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:28.669962    4874 scope.go:122] "RemoveContainer" containerID="0c21bad86504624583c3386731842623b64602c0a4b1e7a3d7331545021f642b"
Oct 30 05:07:28.670191 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:28.669999    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"]
Oct 30 05:07:28.670366 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:07:28.670347    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c21bad86504624583c3386731842623b64602c0a4b1e7a3d7331545021f642b\": container with ID starting with 0c21bad86504624583c3386731842623b64602c0a4b1e7a3d7331545021f642b not found: ID does not exist" containerID="0c21bad86504624583c3386731842623b64602c0a4b1e7a3d7331545021f642b"
Oct 30 05:07:28.670447 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:28.670375    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c21bad86504624583c3386731842623b64602c0a4b1e7a3d7331545021f642b"} err="failed to get container status \"0c21bad86504624583c3386731842623b64602c0a4b1e7a3d7331545021f642b\": rpc error: code = NotFound desc = could not find container \"0c21bad86504624583c3386731842623b64602c0a4b1e7a3d7331545021f642b\": container with ID starting with 0c21bad86504624583c3386731842623b64602c0a4b1e7a3d7331545021f642b not found: ID does not exist"
Oct 30 05:07:28.673463 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:28.673444    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e592d7-openshift-default-6465bbfb69-gcvt6"]
Oct 30 05:07:28.729359 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:28.729337    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ca71311-3f21-47db-98a2-a9f95fbdebe9" path="/var/lib/kubelet/pods/8ca71311-3f21-47db-98a2-a9f95fbdebe9/volumes"
Oct 30 05:07:33.377934 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.377902    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"]
Oct 30 05:07:33.378503 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.378488    4874 cpu_manager.go:414] "containerMap: removing container" podUID="8ca71311-3f21-47db-98a2-a9f95fbdebe9" containerName="istio-proxy"
Oct 30 05:07:33.378552 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.378504    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ca71311-3f21-47db-98a2-a9f95fbdebe9" containerName="istio-proxy"
Oct 30 05:07:33.378552 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.378526    4874 cpu_manager.go:414] "containerMap: removing container" podUID="761c729c-5767-4b78-b916-0841ad83d9e7" containerName="main"
Oct 30 05:07:33.378552 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.378531    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="761c729c-5767-4b78-b916-0841ad83d9e7" containerName="main"
Oct 30 05:07:33.378552 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.378543    4874 cpu_manager.go:414] "containerMap: removing container" podUID="318e0d5c-3645-4477-8a63-13849cca77fc" containerName="maas-api"
Oct 30 05:07:33.378552 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.378548    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="318e0d5c-3645-4477-8a63-13849cca77fc" containerName="maas-api"
Oct 30 05:07:33.378714 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.378563    4874 cpu_manager.go:414] "containerMap: removing container" podUID="8ca71311-3f21-47db-98a2-a9f95fbdebe9" containerName="istio-proxy"
Oct 30 05:07:33.378714 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.378568    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ca71311-3f21-47db-98a2-a9f95fbdebe9" containerName="istio-proxy"
Oct 30 05:07:33.378714 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.378583    4874 cpu_manager.go:414] "containerMap: removing container" podUID="a9735d04-5cad-4502-8f8b-521b47d5c037" containerName="maas-api"
Oct 30 05:07:33.378714 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.378588    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9735d04-5cad-4502-8f8b-521b47d5c037" containerName="maas-api"
Oct 30 05:07:33.378714 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.378602    4874 cpu_manager.go:414] "containerMap: removing container" podUID="e51a82fb-9ec5-4fd4-92c7-6ae38029739b" containerName="maas-api"
Oct 30 05:07:33.378714 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.378606    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e51a82fb-9ec5-4fd4-92c7-6ae38029739b" containerName="maas-api"
Oct 30 05:07:33.378714 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.378681    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="e51a82fb-9ec5-4fd4-92c7-6ae38029739b" containerName="maas-api"
Oct 30 05:07:33.378714 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.378687    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ca71311-3f21-47db-98a2-a9f95fbdebe9" containerName="istio-proxy"
Oct 30 05:07:33.378714 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.378696    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="761c729c-5767-4b78-b916-0841ad83d9e7" containerName="main"
Oct 30 05:07:33.378714 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.378705    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="318e0d5c-3645-4477-8a63-13849cca77fc" containerName="maas-api"
Oct 30 05:07:33.378714 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.378714    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9735d04-5cad-4502-8f8b-521b47d5c037" containerName="maas-api"
Oct 30 05:07:33.383701 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.383676    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:07:33.404369 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.404342    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-e6b837\"/\"openshift-service-ca.crt\""
Oct 30 05:07:33.404538 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.404519    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-e6b837\"/\"kube-root-ca.crt\""
Oct 30 05:07:33.416027 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.415913    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-e6b837\"/\"default-dockercfg-xskpd\""
Oct 30 05:07:33.416027 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.415990    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-e6b837\"/\"sub-test-model-e6b837-kserve-self-signed-certs\""
Oct 30 05:07:33.418976 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.418954    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"]
Oct 30 05:07:33.453135 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.453109    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-dshm\") pod \"sub-test-model-e6b837-kserve-8694db69c9-w52jw\" (UID: \"d605506d-43ec-447c-b573-cb4512867fec\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:07:33.453227 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.453210    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt5ps\" (UniqueName: \"kubernetes.io/projected/d605506d-43ec-447c-b573-cb4512867fec-kube-api-access-zt5ps\") pod \"sub-test-model-e6b837-kserve-8694db69c9-w52jw\" (UID: \"d605506d-43ec-447c-b573-cb4512867fec\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:07:33.453290 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.453263    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-model-cache\") pod \"sub-test-model-e6b837-kserve-8694db69c9-w52jw\" (UID: \"d605506d-43ec-447c-b573-cb4512867fec\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:07:33.453380 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.453367    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-tmp-dir\") pod \"sub-test-model-e6b837-kserve-8694db69c9-w52jw\" (UID: \"d605506d-43ec-447c-b573-cb4512867fec\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:07:33.453459 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.453446    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-home\") pod \"sub-test-model-e6b837-kserve-8694db69c9-w52jw\" (UID: \"d605506d-43ec-447c-b573-cb4512867fec\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:07:33.453517 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.453501    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d605506d-43ec-447c-b573-cb4512867fec-tls-certs\") pod \"sub-test-model-e6b837-kserve-8694db69c9-w52jw\" (UID: \"d605506d-43ec-447c-b573-cb4512867fec\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:07:33.555281 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.555256    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-dshm\") pod \"sub-test-model-e6b837-kserve-8694db69c9-w52jw\" (UID: \"d605506d-43ec-447c-b573-cb4512867fec\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:07:33.555408 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.555324    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt5ps\" (UniqueName: \"kubernetes.io/projected/d605506d-43ec-447c-b573-cb4512867fec-kube-api-access-zt5ps\") pod \"sub-test-model-e6b837-kserve-8694db69c9-w52jw\" (UID: \"d605506d-43ec-447c-b573-cb4512867fec\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:07:33.555408 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.555363    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-model-cache\") pod \"sub-test-model-e6b837-kserve-8694db69c9-w52jw\" (UID: \"d605506d-43ec-447c-b573-cb4512867fec\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:07:33.555512 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.555452    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-tmp-dir\") pod \"sub-test-model-e6b837-kserve-8694db69c9-w52jw\" (UID: \"d605506d-43ec-447c-b573-cb4512867fec\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:07:33.555582 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.555563    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-home\") pod \"sub-test-model-e6b837-kserve-8694db69c9-w52jw\" (UID: \"d605506d-43ec-447c-b573-cb4512867fec\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:07:33.555666 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.555646    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d605506d-43ec-447c-b573-cb4512867fec-tls-certs\") pod \"sub-test-model-e6b837-kserve-8694db69c9-w52jw\" (UID: \"d605506d-43ec-447c-b573-cb4512867fec\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:07:33.555803 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.555778    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-model-cache\") pod \"sub-test-model-e6b837-kserve-8694db69c9-w52jw\" (UID: \"d605506d-43ec-447c-b573-cb4512867fec\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:07:33.555921 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.555818    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-tmp-dir\") pod \"sub-test-model-e6b837-kserve-8694db69c9-w52jw\" (UID: \"d605506d-43ec-447c-b573-cb4512867fec\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:07:33.556008 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.555986    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-home\") pod \"sub-test-model-e6b837-kserve-8694db69c9-w52jw\" (UID: \"d605506d-43ec-447c-b573-cb4512867fec\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:07:33.557530 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.557510    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-dshm\") pod \"sub-test-model-e6b837-kserve-8694db69c9-w52jw\" (UID: \"d605506d-43ec-447c-b573-cb4512867fec\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:07:33.558041 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.558018    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d605506d-43ec-447c-b573-cb4512867fec-tls-certs\") pod \"sub-test-model-e6b837-kserve-8694db69c9-w52jw\" (UID: \"d605506d-43ec-447c-b573-cb4512867fec\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:07:33.578909 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.578887    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt5ps\" (UniqueName: \"kubernetes.io/projected/d605506d-43ec-447c-b573-cb4512867fec-kube-api-access-zt5ps\") pod \"sub-test-model-e6b837-kserve-8694db69c9-w52jw\" (UID: \"d605506d-43ec-447c-b573-cb4512867fec\") " pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:07:33.699016 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:33.698672    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:07:34.011373 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:34.011338    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"]
Oct 30 05:07:34.012981 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:07:34.012953    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd605506d_43ec_447c_b573_cb4512867fec.slice/crio-3d1165b467f9fae2cfa1565955627de1fac2454ec75a41ecaa7a1c9d45b035e5 WatchSource:0}: Error finding container 3d1165b467f9fae2cfa1565955627de1fac2454ec75a41ecaa7a1c9d45b035e5: Status 404 returned error can't find the container with id 3d1165b467f9fae2cfa1565955627de1fac2454ec75a41ecaa7a1c9d45b035e5
Oct 30 05:07:34.668431 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:34.668396    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw" event={"ID":"d605506d-43ec-447c-b573-cb4512867fec","Type":"ContainerStarted","Data":"551194fb9431963d868b3fae067fef5fed774de8200128553e8c1af23c94a568"}
Oct 30 05:07:34.668849 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:34.668437    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw" event={"ID":"d605506d-43ec-447c-b573-cb4512867fec","Type":"ContainerStarted","Data":"3d1165b467f9fae2cfa1565955627de1fac2454ec75a41ecaa7a1c9d45b035e5"}
Oct 30 05:07:34.699759 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:34.699736    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:07:34.711815 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:34.711772    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw" podStartSLOduration=1.711760522 podStartE2EDuration="1.711760522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:07:34.70855353 +0000 UTC m=+4174.120949102" watchObservedRunningTime="2026-10-30 05:07:34.711760522 +0000 UTC m=+4174.124156095"
Oct 30 05:07:43.699616 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:43.699529    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:07:43.713355 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:43.713330    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:07:43.770196 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:43.770165    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:07:44.065842 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:44.065812    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"]
Oct 30 05:07:44.066040 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:44.066015    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q" podUID="dc7b1652-bcc3-412b-b416-3fecd1ec9b90" containerName="istio-proxy" containerID="cri-o://6e4388c5246c264747eed1146ab8749a065952ad519e10b35a84ff8974de9355" gracePeriod=30
Oct 30 05:07:44.856036 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:44.856005    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:07:49.759675 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.759651    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 05:07:49.840287 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.840220    4874 generic.go:356] "Generic (PLEG): container finished" podID="dc7b1652-bcc3-412b-b416-3fecd1ec9b90" containerID="6e4388c5246c264747eed1146ab8749a065952ad519e10b35a84ff8974de9355" exitCode=0
Oct 30 05:07:49.840488 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.840300    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q" event={"ID":"dc7b1652-bcc3-412b-b416-3fecd1ec9b90","Type":"ContainerDied","Data":"6e4388c5246c264747eed1146ab8749a065952ad519e10b35a84ff8974de9355"}
Oct 30 05:07:49.840488 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.840314    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"
Oct 30 05:07:49.840488 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.840405    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q" event={"ID":"dc7b1652-bcc3-412b-b416-3fecd1ec9b90","Type":"ContainerDied","Data":"6ae7ffd41cb039ddbe1e2851774d8d49d09e0017807ebacd5d9e596421e8ff99"}
Oct 30 05:07:49.840488 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.840438    4874 scope.go:122] "RemoveContainer" containerID="6e4388c5246c264747eed1146ab8749a065952ad519e10b35a84ff8974de9355"
Oct 30 05:07:49.853300 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.853275    4874 scope.go:122] "RemoveContainer" containerID="6e4388c5246c264747eed1146ab8749a065952ad519e10b35a84ff8974de9355"
Oct 30 05:07:49.853645 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:07:49.853623    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e4388c5246c264747eed1146ab8749a065952ad519e10b35a84ff8974de9355\": container with ID starting with 6e4388c5246c264747eed1146ab8749a065952ad519e10b35a84ff8974de9355 not found: ID does not exist" containerID="6e4388c5246c264747eed1146ab8749a065952ad519e10b35a84ff8974de9355"
Oct 30 05:07:49.853716 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.853655    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e4388c5246c264747eed1146ab8749a065952ad519e10b35a84ff8974de9355"} err="failed to get container status \"6e4388c5246c264747eed1146ab8749a065952ad519e10b35a84ff8974de9355\": rpc error: code = NotFound desc = could not find container \"6e4388c5246c264747eed1146ab8749a065952ad519e10b35a84ff8974de9355\": container with ID starting with 6e4388c5246c264747eed1146ab8749a065952ad519e10b35a84ff8974de9355 not found: ID does not exist"
Oct 30 05:07:49.887666 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.887580    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-envoy\") pod \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") "
Oct 30 05:07:49.887666 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.887629    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-workload-certs\") pod \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") "
Oct 30 05:07:49.887883 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.887683    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-token\" (UniqueName: \"kubernetes.io/projected/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-token\") pod \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") "
Oct 30 05:07:49.887883 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.887716    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-data\") pod \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") "
Oct 30 05:07:49.887883 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.887744    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istiod-ca-cert\") pod \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") "
Oct 30 05:07:49.887883 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.887803    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-workload-socket\") pod \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") "
Oct 30 05:07:49.887883 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.887843    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-podinfo\") pod \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") "
Oct 30 05:07:49.888135 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.887944    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-kube-api-access-lxm7m\" (UniqueName: \"kubernetes.io/projected/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-kube-api-access-lxm7m\") pod \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") "
Oct 30 05:07:49.888135 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.887944    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-workload-certs" pod "dc7b1652-bcc3-412b-b416-3fecd1ec9b90" (UID: "dc7b1652-bcc3-412b-b416-3fecd1ec9b90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:07:49.888135 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.888010    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-data" pod "dc7b1652-bcc3-412b-b416-3fecd1ec9b90" (UID: "dc7b1652-bcc3-412b-b416-3fecd1ec9b90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:07:49.888325 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.888134    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-credential-socket\") pod \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\" (UID: \"dc7b1652-bcc3-412b-b416-3fecd1ec9b90\") "
Oct 30 05:07:49.888325 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.888134    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-workload-socket" pod "dc7b1652-bcc3-412b-b416-3fecd1ec9b90" (UID: "dc7b1652-bcc3-412b-b416-3fecd1ec9b90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:07:49.888325 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.888283    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istiod-ca-cert" pod "dc7b1652-bcc3-412b-b416-3fecd1ec9b90" (UID: "dc7b1652-bcc3-412b-b416-3fecd1ec9b90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:07:49.888474 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.888455    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-credential-socket" pod "dc7b1652-bcc3-412b-b416-3fecd1ec9b90" (UID: "dc7b1652-bcc3-412b-b416-3fecd1ec9b90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:07:49.888768 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.888746    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:49.888824 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.888777    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:49.888824 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.888794    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:49.888824 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.888808    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:49.888824 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.888820    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:49.890140 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.890119    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-kube-api-access-lxm7m" pod "dc7b1652-bcc3-412b-b416-3fecd1ec9b90" (UID: "dc7b1652-bcc3-412b-b416-3fecd1ec9b90"). InnerVolumeSpecName "kube-api-access-lxm7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:07:49.890225 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.890159    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-envoy" pod "dc7b1652-bcc3-412b-b416-3fecd1ec9b90" (UID: "dc7b1652-bcc3-412b-b416-3fecd1ec9b90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:07:49.890444 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.890420    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-podinfo" pod "dc7b1652-bcc3-412b-b416-3fecd1ec9b90" (UID: "dc7b1652-bcc3-412b-b416-3fecd1ec9b90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:07:49.890444 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.890433    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-token" pod "dc7b1652-bcc3-412b-b416-3fecd1ec9b90" (UID: "dc7b1652-bcc3-412b-b416-3fecd1ec9b90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:07:49.989770 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.989735    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:49.989770 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.989766    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:49.989770 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.989775    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:49.990017 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:49.989784    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxm7m\" (UniqueName: \"kubernetes.io/projected/dc7b1652-bcc3-412b-b416-3fecd1ec9b90-kube-api-access-lxm7m\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:07:50.186968 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:50.186939    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"]
Oct 30 05:07:50.198130 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:50.198105    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-m2l5q"]
Oct 30 05:07:50.773262 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:50.767912    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc7b1652-bcc3-412b-b416-3fecd1ec9b90" path="/var/lib/kubelet/pods/dc7b1652-bcc3-412b-b416-3fecd1ec9b90/volumes"
Oct 30 05:07:55.205565 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.205527    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"]
Oct 30 05:07:55.206513 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.206491    4874 cpu_manager.go:414] "containerMap: removing container" podUID="dc7b1652-bcc3-412b-b416-3fecd1ec9b90" containerName="istio-proxy"
Oct 30 05:07:55.206626 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.206515    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc7b1652-bcc3-412b-b416-3fecd1ec9b90" containerName="istio-proxy"
Oct 30 05:07:55.206696 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.206684    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ca71311-3f21-47db-98a2-a9f95fbdebe9" containerName="istio-proxy"
Oct 30 05:07:55.206733 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.206707    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc7b1652-bcc3-412b-b416-3fecd1ec9b90" containerName="istio-proxy"
Oct 30 05:07:55.214921 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.214307    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:07:55.238017 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.237989    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"]
Oct 30 05:07:55.245929 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.245903    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-247158\"/\"sub-test-model-247158-kserve-self-signed-certs\""
Oct 30 05:07:55.246059 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.245966    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-247158\"/\"default-dockercfg-g7h4m\""
Oct 30 05:07:55.250591 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.250571    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-247158\"/\"openshift-service-ca.crt\""
Oct 30 05:07:55.250591 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.250588    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-247158\"/\"kube-root-ca.crt\""
Oct 30 05:07:55.378833 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.378799    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-home\") pod \"sub-test-model-247158-kserve-c47cff-qqs9x\" (UID: \"d363d951-949c-4438-8c54-e65908025cfd\") " pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:07:55.378998 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.378852    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d363d951-949c-4438-8c54-e65908025cfd-tls-certs\") pod \"sub-test-model-247158-kserve-c47cff-qqs9x\" (UID: \"d363d951-949c-4438-8c54-e65908025cfd\") " pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:07:55.378998 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.378973    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc22b\" (UniqueName: \"kubernetes.io/projected/d363d951-949c-4438-8c54-e65908025cfd-kube-api-access-lc22b\") pod \"sub-test-model-247158-kserve-c47cff-qqs9x\" (UID: \"d363d951-949c-4438-8c54-e65908025cfd\") " pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:07:55.379074 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.379024    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-tmp-dir\") pod \"sub-test-model-247158-kserve-c47cff-qqs9x\" (UID: \"d363d951-949c-4438-8c54-e65908025cfd\") " pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:07:55.379165 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.379151    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-dshm\") pod \"sub-test-model-247158-kserve-c47cff-qqs9x\" (UID: \"d363d951-949c-4438-8c54-e65908025cfd\") " pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:07:55.379248 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.379220    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-model-cache\") pod \"sub-test-model-247158-kserve-c47cff-qqs9x\" (UID: \"d363d951-949c-4438-8c54-e65908025cfd\") " pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:07:55.490100 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.490059    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-tmp-dir\") pod \"sub-test-model-247158-kserve-c47cff-qqs9x\" (UID: \"d363d951-949c-4438-8c54-e65908025cfd\") " pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:07:55.490495 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.490466    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-dshm\") pod \"sub-test-model-247158-kserve-c47cff-qqs9x\" (UID: \"d363d951-949c-4438-8c54-e65908025cfd\") " pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:07:55.490710 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.490693    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-model-cache\") pod \"sub-test-model-247158-kserve-c47cff-qqs9x\" (UID: \"d363d951-949c-4438-8c54-e65908025cfd\") " pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:07:55.490867 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.490851    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-home\") pod \"sub-test-model-247158-kserve-c47cff-qqs9x\" (UID: \"d363d951-949c-4438-8c54-e65908025cfd\") " pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:07:55.491011 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.490996    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d363d951-949c-4438-8c54-e65908025cfd-tls-certs\") pod \"sub-test-model-247158-kserve-c47cff-qqs9x\" (UID: \"d363d951-949c-4438-8c54-e65908025cfd\") " pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:07:55.491958 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.491936    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc22b\" (UniqueName: \"kubernetes.io/projected/d363d951-949c-4438-8c54-e65908025cfd-kube-api-access-lc22b\") pod \"sub-test-model-247158-kserve-c47cff-qqs9x\" (UID: \"d363d951-949c-4438-8c54-e65908025cfd\") " pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:07:55.492822 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.492796    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-model-cache\") pod \"sub-test-model-247158-kserve-c47cff-qqs9x\" (UID: \"d363d951-949c-4438-8c54-e65908025cfd\") " pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:07:55.493224 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.493204    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-home\") pod \"sub-test-model-247158-kserve-c47cff-qqs9x\" (UID: \"d363d951-949c-4438-8c54-e65908025cfd\") " pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:07:55.494059 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.494034    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-tmp-dir\") pod \"sub-test-model-247158-kserve-c47cff-qqs9x\" (UID: \"d363d951-949c-4438-8c54-e65908025cfd\") " pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:07:55.530282 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.523727    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-dshm\") pod \"sub-test-model-247158-kserve-c47cff-qqs9x\" (UID: \"d363d951-949c-4438-8c54-e65908025cfd\") " pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:07:55.530517 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.527283    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc22b\" (UniqueName: \"kubernetes.io/projected/d363d951-949c-4438-8c54-e65908025cfd-kube-api-access-lc22b\") pod \"sub-test-model-247158-kserve-c47cff-qqs9x\" (UID: \"d363d951-949c-4438-8c54-e65908025cfd\") " pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:07:55.560345 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.560310    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d363d951-949c-4438-8c54-e65908025cfd-tls-certs\") pod \"sub-test-model-247158-kserve-c47cff-qqs9x\" (UID: \"d363d951-949c-4438-8c54-e65908025cfd\") " pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:07:55.677918 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:07:55.677881    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:07:55.834764 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.834678    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:07:55.997028 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.996994    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"]
Oct 30 05:07:55.997225 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:55.997200    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj" podUID="54b0d7fa-4b81-49a9-9819-2ee76bd5177e" containerName="istio-proxy" containerID="cri-o://f5cc58b6414b48fca091651845018b80bfd426adc58ad9872872ed5a0ada161e" gracePeriod=30
Oct 30 05:07:56.168676 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:56.168584    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"]
Oct 30 05:07:56.208306 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:07:56.193801    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd363d951_949c_4438_8c54_e65908025cfd.slice/crio-d6114bb3f10a8c3826ce42dba1eeb9b177753d7445be38701f058a7199bffaea WatchSource:0}: Error finding container d6114bb3f10a8c3826ce42dba1eeb9b177753d7445be38701f058a7199bffaea: Status 404 returned error can't find the container with id d6114bb3f10a8c3826ce42dba1eeb9b177753d7445be38701f058a7199bffaea
Oct 30 05:07:56.891960 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:56.891922    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x" event={"ID":"d363d951-949c-4438-8c54-e65908025cfd","Type":"ContainerStarted","Data":"fde13ad63db0aa367fdd79f947bbf8a601b2f15e65b891773463fa1acc0fbbbc"}
Oct 30 05:07:56.891960 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:56.891965    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x" event={"ID":"d363d951-949c-4438-8c54-e65908025cfd","Type":"ContainerStarted","Data":"d6114bb3f10a8c3826ce42dba1eeb9b177753d7445be38701f058a7199bffaea"}
Oct 30 05:07:56.932097 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:56.932045    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x" podStartSLOduration=1.932031046 podStartE2EDuration="1.932031046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:07:56.93099904 +0000 UTC m=+4196.343394613" watchObservedRunningTime="2026-10-30 05:07:56.932031046 +0000 UTC m=+4196.344426618"
Oct 30 05:07:57.834798 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:07:57.834759    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:08:01.270255 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.270176    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 05:08:01.341896 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.340312    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 05:08:01.346565 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.346538    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 05:08:01.368552 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.368528    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-envoy\") pod \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") "
Oct 30 05:08:01.368695 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.368577    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-token\" (UniqueName: \"kubernetes.io/projected/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-token\") pod \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") "
Oct 30 05:08:01.368695 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.368681    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-workload-certs\") pod \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") "
Oct 30 05:08:01.368788 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.368737    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-workload-socket\") pod \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") "
Oct 30 05:08:01.368845 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.368784    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-kube-api-access-j4jsg\" (UniqueName: \"kubernetes.io/projected/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-kube-api-access-j4jsg\") pod \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") "
Oct 30 05:08:01.368845 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.368821    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-podinfo\") pod \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") "
Oct 30 05:08:01.368956 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.368894    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-data\") pod \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") "
Oct 30 05:08:01.369016 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.368963    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-credential-socket\") pod \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") "
Oct 30 05:08:01.369016 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.368987    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-workload-certs" pod "54b0d7fa-4b81-49a9-9819-2ee76bd5177e" (UID: "54b0d7fa-4b81-49a9-9819-2ee76bd5177e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:08:01.369016 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.369012    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istiod-ca-cert\") pod \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\" (UID: \"54b0d7fa-4b81-49a9-9819-2ee76bd5177e\") "
Oct 30 05:08:01.369285 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.369035    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-workload-socket" pod "54b0d7fa-4b81-49a9-9819-2ee76bd5177e" (UID: "54b0d7fa-4b81-49a9-9819-2ee76bd5177e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:08:01.369285 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.369214    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-data" pod "54b0d7fa-4b81-49a9-9819-2ee76bd5177e" (UID: "54b0d7fa-4b81-49a9-9819-2ee76bd5177e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:08:01.369405 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.369296    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-credential-socket" pod "54b0d7fa-4b81-49a9-9819-2ee76bd5177e" (UID: "54b0d7fa-4b81-49a9-9819-2ee76bd5177e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:08:01.369558 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.369532    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istiod-ca-cert" pod "54b0d7fa-4b81-49a9-9819-2ee76bd5177e" (UID: "54b0d7fa-4b81-49a9-9819-2ee76bd5177e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:08:01.369683 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.369663    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:08:01.369729 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.369687    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:08:01.369729 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.369701    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:08:01.369729 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.369718    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:08:01.369830 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.369730    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:08:01.371354 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.371330    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-kube-api-access-j4jsg" pod "54b0d7fa-4b81-49a9-9819-2ee76bd5177e" (UID: "54b0d7fa-4b81-49a9-9819-2ee76bd5177e"). InnerVolumeSpecName "kube-api-access-j4jsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:08:01.371825 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.371794    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-envoy" pod "54b0d7fa-4b81-49a9-9819-2ee76bd5177e" (UID: "54b0d7fa-4b81-49a9-9819-2ee76bd5177e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:08:01.371825 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.371804    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-token" pod "54b0d7fa-4b81-49a9-9819-2ee76bd5177e" (UID: "54b0d7fa-4b81-49a9-9819-2ee76bd5177e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:08:01.371993 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.371876    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-podinfo" pod "54b0d7fa-4b81-49a9-9819-2ee76bd5177e" (UID: "54b0d7fa-4b81-49a9-9819-2ee76bd5177e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:08:01.471414 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.471380    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4jsg\" (UniqueName: \"kubernetes.io/projected/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-kube-api-access-j4jsg\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:08:01.471414 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.471408    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:08:01.471414 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.471417    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:08:01.471643 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.471426    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b0d7fa-4b81-49a9-9819-2ee76bd5177e-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:08:01.938166 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.938138    4874 generic.go:356] "Generic (PLEG): container finished" podID="54b0d7fa-4b81-49a9-9819-2ee76bd5177e" containerID="f5cc58b6414b48fca091651845018b80bfd426adc58ad9872872ed5a0ada161e" exitCode=0
Oct 30 05:08:01.938367 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.938186    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"
Oct 30 05:08:01.938367 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.938202    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj" event={"ID":"54b0d7fa-4b81-49a9-9819-2ee76bd5177e","Type":"ContainerDied","Data":"f5cc58b6414b48fca091651845018b80bfd426adc58ad9872872ed5a0ada161e"}
Oct 30 05:08:01.938367 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.938258    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj" event={"ID":"54b0d7fa-4b81-49a9-9819-2ee76bd5177e","Type":"ContainerDied","Data":"0ef6185afc8e816984dbabcfdd497bce5e2c1ae7a5802a0356a346c4a92a4c31"}
Oct 30 05:08:01.938367 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.938290    4874 scope.go:122] "RemoveContainer" containerID="f5cc58b6414b48fca091651845018b80bfd426adc58ad9872872ed5a0ada161e"
Oct 30 05:08:01.949033 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.949017    4874 scope.go:122] "RemoveContainer" containerID="08779a31008c91db6218ef983c33d6da7b733e9c2ca22fbba82ba214e54b87f1"
Oct 30 05:08:01.958145 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.958123    4874 scope.go:122] "RemoveContainer" containerID="f5cc58b6414b48fca091651845018b80bfd426adc58ad9872872ed5a0ada161e"
Oct 30 05:08:01.958431 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:08:01.958413    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5cc58b6414b48fca091651845018b80bfd426adc58ad9872872ed5a0ada161e\": container with ID starting with f5cc58b6414b48fca091651845018b80bfd426adc58ad9872872ed5a0ada161e not found: ID does not exist" containerID="f5cc58b6414b48fca091651845018b80bfd426adc58ad9872872ed5a0ada161e"
Oct 30 05:08:01.958500 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.958440    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5cc58b6414b48fca091651845018b80bfd426adc58ad9872872ed5a0ada161e"} err="failed to get container status \"f5cc58b6414b48fca091651845018b80bfd426adc58ad9872872ed5a0ada161e\": rpc error: code = NotFound desc = could not find container \"f5cc58b6414b48fca091651845018b80bfd426adc58ad9872872ed5a0ada161e\": container with ID starting with f5cc58b6414b48fca091651845018b80bfd426adc58ad9872872ed5a0ada161e not found: ID does not exist"
Oct 30 05:08:01.958500 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.958458    4874 scope.go:122] "RemoveContainer" containerID="08779a31008c91db6218ef983c33d6da7b733e9c2ca22fbba82ba214e54b87f1"
Oct 30 05:08:01.958669 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:08:01.958650    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08779a31008c91db6218ef983c33d6da7b733e9c2ca22fbba82ba214e54b87f1\": container with ID starting with 08779a31008c91db6218ef983c33d6da7b733e9c2ca22fbba82ba214e54b87f1 not found: ID does not exist" containerID="08779a31008c91db6218ef983c33d6da7b733e9c2ca22fbba82ba214e54b87f1"
Oct 30 05:08:01.958707 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.958672    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08779a31008c91db6218ef983c33d6da7b733e9c2ca22fbba82ba214e54b87f1"} err="failed to get container status \"08779a31008c91db6218ef983c33d6da7b733e9c2ca22fbba82ba214e54b87f1\": rpc error: code = NotFound desc = could not find container \"08779a31008c91db6218ef983c33d6da7b733e9c2ca22fbba82ba214e54b87f1\": container with ID starting with 08779a31008c91db6218ef983c33d6da7b733e9c2ca22fbba82ba214e54b87f1 not found: ID does not exist"
Oct 30 05:08:01.988368 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.988342    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"]
Oct 30 05:08:01.999478 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:01.999449    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-26b72f-openshift-default-7b644d5555-j45zj"]
Oct 30 05:08:02.730643 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:02.730608    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54b0d7fa-4b81-49a9-9819-2ee76bd5177e" path="/var/lib/kubelet/pods/54b0d7fa-4b81-49a9-9819-2ee76bd5177e/volumes"
Oct 30 05:08:05.835290 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:05.835216    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:08:05.848741 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:05.848712    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:08:05.974357 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:05.974333    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:08:25.610348 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:08:25.610314    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:08:33.383398 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:08:33.375564    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:08:55.610700 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:08:55.610665    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:09:15.274832 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:09:15.274750    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm"]
Oct 30 05:09:15.275420 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:09:15.274933    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm" podUID="2bea58a5-bdac-4545-9aee-66961d2bd86b" containerName="maas-api" containerID="cri-o://de154c6a5a5a25159626453a9711f9d708e4e4735014da3372091d6d4cc8e781" gracePeriod=30
Oct 30 05:09:15.512897 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:09:15.512712    4874 generic.go:356] "Generic (PLEG): container finished" podID="2bea58a5-bdac-4545-9aee-66961d2bd86b" containerID="de154c6a5a5a25159626453a9711f9d708e4e4735014da3372091d6d4cc8e781" exitCode=0
Oct 30 05:09:15.512897 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:09:15.512830    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm" event={"ID":"2bea58a5-bdac-4545-9aee-66961d2bd86b","Type":"ContainerDied","Data":"de154c6a5a5a25159626453a9711f9d708e4e4735014da3372091d6d4cc8e781"}
Oct 30 05:09:15.632758 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:09:15.632732    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm"
Oct 30 05:09:15.760420 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:09:15.760376    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bea58a5-bdac-4545-9aee-66961d2bd86b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bea58a5-bdac-4545-9aee-66961d2bd86b-openshift-service-ca\") pod \"2bea58a5-bdac-4545-9aee-66961d2bd86b\" (UID: \"2bea58a5-bdac-4545-9aee-66961d2bd86b\") "
Oct 30 05:09:15.784140 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:09:15.784110    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bea58a5-bdac-4545-9aee-66961d2bd86b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bea58a5-bdac-4545-9aee-66961d2bd86b-metrics-certs\") pod \"2bea58a5-bdac-4545-9aee-66961d2bd86b\" (UID: \"2bea58a5-bdac-4545-9aee-66961d2bd86b\") "
Oct 30 05:09:15.789868 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:09:15.789839    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bea58a5-bdac-4545-9aee-66961d2bd86b-kube-api-access-gr48c\" (UniqueName: \"kubernetes.io/projected/2bea58a5-bdac-4545-9aee-66961d2bd86b-kube-api-access-gr48c\") pod \"2bea58a5-bdac-4545-9aee-66961d2bd86b\" (UID: \"2bea58a5-bdac-4545-9aee-66961d2bd86b\") "
Oct 30 05:09:15.790162 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:09:15.790143    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bea58a5-bdac-4545-9aee-66961d2bd86b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bea58a5-bdac-4545-9aee-66961d2bd86b-maas-api-tls\") pod \"2bea58a5-bdac-4545-9aee-66961d2bd86b\" (UID: \"2bea58a5-bdac-4545-9aee-66961d2bd86b\") "
Oct 30 05:09:15.791627 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:09:15.767446    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bea58a5-bdac-4545-9aee-66961d2bd86b-openshift-service-ca" pod "2bea58a5-bdac-4545-9aee-66961d2bd86b" (UID: "2bea58a5-bdac-4545-9aee-66961d2bd86b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:09:15.810803 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:09:15.810743    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bea58a5-bdac-4545-9aee-66961d2bd86b-kube-api-access-gr48c" pod "2bea58a5-bdac-4545-9aee-66961d2bd86b" (UID: "2bea58a5-bdac-4545-9aee-66961d2bd86b"). InnerVolumeSpecName "kube-api-access-gr48c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:09:15.811173 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:09:15.811132    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bea58a5-bdac-4545-9aee-66961d2bd86b-metrics-certs" pod "2bea58a5-bdac-4545-9aee-66961d2bd86b" (UID: "2bea58a5-bdac-4545-9aee-66961d2bd86b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:09:15.813737 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:09:15.813716    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bea58a5-bdac-4545-9aee-66961d2bd86b-maas-api-tls" pod "2bea58a5-bdac-4545-9aee-66961d2bd86b" (UID: "2bea58a5-bdac-4545-9aee-66961d2bd86b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:09:15.892777 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:09:15.892686    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bea58a5-bdac-4545-9aee-66961d2bd86b-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:09:15.892987 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:09:15.892967    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr48c\" (UniqueName: \"kubernetes.io/projected/2bea58a5-bdac-4545-9aee-66961d2bd86b-kube-api-access-gr48c\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:09:15.893059 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:09:15.892994    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bea58a5-bdac-4545-9aee-66961d2bd86b-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:09:15.893059 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:09:15.893009    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bea58a5-bdac-4545-9aee-66961d2bd86b-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:09:16.520140 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:09:16.520110    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm"
Oct 30 05:09:16.520587 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:09:16.520152    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm" event={"ID":"2bea58a5-bdac-4545-9aee-66961d2bd86b","Type":"ContainerDied","Data":"9d04545a4456ce87c3045645cfaf9bada668a64a875bc8171c505795aa5b825c"}
Oct 30 05:09:16.520587 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:09:16.520209    4874 scope.go:122] "RemoveContainer" containerID="de154c6a5a5a25159626453a9711f9d708e4e4735014da3372091d6d4cc8e781"
Oct 30 05:09:16.569671 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:09:16.569639    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm"]
Oct 30 05:09:16.580917 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:09:16.580892    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ed8266-7db4bb59d5-bldqm"]
Oct 30 05:09:16.729808 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:09:16.729776    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bea58a5-bdac-4545-9aee-66961d2bd86b" path="/var/lib/kubelet/pods/2bea58a5-bdac-4545-9aee-66961d2bd86b/volumes"
Oct 30 05:09:17.423798 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:09:17.423762    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:09:25.609096 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:09:25.609060    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:09:50.300569 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:09:50.300528    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:09:55.610071 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:09:55.610037    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:10:00.101894 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:00.101858    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h"]
Oct 30 05:10:00.102627 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:00.102107    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h" podUID="70bb7edd-3719-4c86-b358-2039dbd00675" containerName="maas-api" containerID="cri-o://2e88f8afd6aab7da47be79c82173743943faa1c7a09921870bbe3c8a37de0394" gracePeriod=30
Oct 30 05:10:00.447311 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:00.447285    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h"
Oct 30 05:10:00.524264 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:00.523677    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70bb7edd-3719-4c86-b358-2039dbd00675-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70bb7edd-3719-4c86-b358-2039dbd00675-maas-api-tls\") pod \"70bb7edd-3719-4c86-b358-2039dbd00675\" (UID: \"70bb7edd-3719-4c86-b358-2039dbd00675\") "
Oct 30 05:10:00.524264 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:00.523891    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70bb7edd-3719-4c86-b358-2039dbd00675-metrics-certs\" (UniqueName: \"kubernetes.io/secret/70bb7edd-3719-4c86-b358-2039dbd00675-metrics-certs\") pod \"70bb7edd-3719-4c86-b358-2039dbd00675\" (UID: \"70bb7edd-3719-4c86-b358-2039dbd00675\") "
Oct 30 05:10:00.524264 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:00.523936    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70bb7edd-3719-4c86-b358-2039dbd00675-kube-api-access-wsxtj\" (UniqueName: \"kubernetes.io/projected/70bb7edd-3719-4c86-b358-2039dbd00675-kube-api-access-wsxtj\") pod \"70bb7edd-3719-4c86-b358-2039dbd00675\" (UID: \"70bb7edd-3719-4c86-b358-2039dbd00675\") "
Oct 30 05:10:00.524264 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:00.523969    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70bb7edd-3719-4c86-b358-2039dbd00675-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70bb7edd-3719-4c86-b358-2039dbd00675-openshift-service-ca\") pod \"70bb7edd-3719-4c86-b358-2039dbd00675\" (UID: \"70bb7edd-3719-4c86-b358-2039dbd00675\") "
Oct 30 05:10:00.528616 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:00.528583    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70bb7edd-3719-4c86-b358-2039dbd00675-openshift-service-ca" pod "70bb7edd-3719-4c86-b358-2039dbd00675" (UID: "70bb7edd-3719-4c86-b358-2039dbd00675"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:10:00.545585 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:00.545539    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70bb7edd-3719-4c86-b358-2039dbd00675-kube-api-access-wsxtj" pod "70bb7edd-3719-4c86-b358-2039dbd00675" (UID: "70bb7edd-3719-4c86-b358-2039dbd00675"). InnerVolumeSpecName "kube-api-access-wsxtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:10:00.550043 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:00.550010    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70bb7edd-3719-4c86-b358-2039dbd00675-metrics-certs" pod "70bb7edd-3719-4c86-b358-2039dbd00675" (UID: "70bb7edd-3719-4c86-b358-2039dbd00675"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:10:00.550205 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:00.550053    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70bb7edd-3719-4c86-b358-2039dbd00675-maas-api-tls" pod "70bb7edd-3719-4c86-b358-2039dbd00675" (UID: "70bb7edd-3719-4c86-b358-2039dbd00675"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:10:00.650328 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:00.644822    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70bb7edd-3719-4c86-b358-2039dbd00675-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:00.650328 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:00.644854    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70bb7edd-3719-4c86-b358-2039dbd00675-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:00.650328 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:00.644871    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsxtj\" (UniqueName: \"kubernetes.io/projected/70bb7edd-3719-4c86-b358-2039dbd00675-kube-api-access-wsxtj\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:00.650328 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:00.644887    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70bb7edd-3719-4c86-b358-2039dbd00675-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:00.878153 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:00.874409    4874 generic.go:356] "Generic (PLEG): container finished" podID="70bb7edd-3719-4c86-b358-2039dbd00675" containerID="2e88f8afd6aab7da47be79c82173743943faa1c7a09921870bbe3c8a37de0394" exitCode=0
Oct 30 05:10:00.878153 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:00.874543    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h" event={"ID":"70bb7edd-3719-4c86-b358-2039dbd00675","Type":"ContainerDied","Data":"2e88f8afd6aab7da47be79c82173743943faa1c7a09921870bbe3c8a37de0394"}
Oct 30 05:10:00.878153 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:00.874607    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h" event={"ID":"70bb7edd-3719-4c86-b358-2039dbd00675","Type":"ContainerDied","Data":"f5158b8dcfe9380dd48e1db745ba9bbff70b8ca36489d10434b104faf116fcde"}
Oct 30 05:10:00.878153 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:00.874647    4874 scope.go:122] "RemoveContainer" containerID="2e88f8afd6aab7da47be79c82173743943faa1c7a09921870bbe3c8a37de0394"
Oct 30 05:10:00.878153 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:00.874887    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h"
Oct 30 05:10:00.899506 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:00.899474    4874 scope.go:122] "RemoveContainer" containerID="2e88f8afd6aab7da47be79c82173743943faa1c7a09921870bbe3c8a37de0394"
Oct 30 05:10:00.900074 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:10:00.900016    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e88f8afd6aab7da47be79c82173743943faa1c7a09921870bbe3c8a37de0394\": container with ID starting with 2e88f8afd6aab7da47be79c82173743943faa1c7a09921870bbe3c8a37de0394 not found: ID does not exist" containerID="2e88f8afd6aab7da47be79c82173743943faa1c7a09921870bbe3c8a37de0394"
Oct 30 05:10:00.900276 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:00.900222    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e88f8afd6aab7da47be79c82173743943faa1c7a09921870bbe3c8a37de0394"} err="failed to get container status \"2e88f8afd6aab7da47be79c82173743943faa1c7a09921870bbe3c8a37de0394\": rpc error: code = NotFound desc = could not find container \"2e88f8afd6aab7da47be79c82173743943faa1c7a09921870bbe3c8a37de0394\": container with ID starting with 2e88f8afd6aab7da47be79c82173743943faa1c7a09921870bbe3c8a37de0394 not found: ID does not exist"
Oct 30 05:10:00.934530 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:00.934473    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h"]
Oct 30 05:10:00.945495 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:00.945472    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6378cc-7cc96b694d-k2h8h"]
Oct 30 05:10:02.731044 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:02.731010    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70bb7edd-3719-4c86-b358-2039dbd00675" path="/var/lib/kubelet/pods/70bb7edd-3719-4c86-b358-2039dbd00675/volumes"
Oct 30 05:10:25.610183 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:10:25.610142    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:10:26.915354 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:26.915320    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"]
Oct 30 05:10:26.915781 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:26.915564    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p" podUID="c89aeca9-c4fc-4f9c-ba74-033b72349ddf" containerName="istio-proxy" containerID="cri-o://5efff0f058281139942ad6c2ab4fd98d67181a178e1b3d8cd72658c74a1475f5" gracePeriod=30
Oct 30 05:10:32.155797 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.155758    4874 generic.go:356] "Generic (PLEG): container finished" podID="c89aeca9-c4fc-4f9c-ba74-033b72349ddf" containerID="5efff0f058281139942ad6c2ab4fd98d67181a178e1b3d8cd72658c74a1475f5" exitCode=0
Oct 30 05:10:32.156166 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.155799    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p" event={"ID":"c89aeca9-c4fc-4f9c-ba74-033b72349ddf","Type":"ContainerDied","Data":"5efff0f058281139942ad6c2ab4fd98d67181a178e1b3d8cd72658c74a1475f5"}
Oct 30 05:10:32.200937 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.200914    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 05:10:32.262012 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.261982    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-podinfo\") pod \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") "
Oct 30 05:10:32.262167 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.262037    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-token\" (UniqueName: \"kubernetes.io/projected/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-token\") pod \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") "
Oct 30 05:10:32.262266 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.262198    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-workload-socket\") pod \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") "
Oct 30 05:10:32.262345 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.262272    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-workload-certs\") pod \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") "
Oct 30 05:10:32.262345 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.262333    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-envoy\") pod \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") "
Oct 30 05:10:32.262459 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.262364    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-kube-api-access-j4sfd\" (UniqueName: \"kubernetes.io/projected/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-kube-api-access-j4sfd\") pod \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") "
Oct 30 05:10:32.262518 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.262494    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-workload-certs" pod "c89aeca9-c4fc-4f9c-ba74-033b72349ddf" (UID: "c89aeca9-c4fc-4f9c-ba74-033b72349ddf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:10:32.262581 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.262518    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-workload-socket" pod "c89aeca9-c4fc-4f9c-ba74-033b72349ddf" (UID: "c89aeca9-c4fc-4f9c-ba74-033b72349ddf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:10:32.263046 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.263025    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istiod-ca-cert\") pod \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") "
Oct 30 05:10:32.263699 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.263678    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-data\") pod \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") "
Oct 30 05:10:32.263909 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.263889    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-credential-socket\") pod \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\" (UID: \"c89aeca9-c4fc-4f9c-ba74-033b72349ddf\") "
Oct 30 05:10:32.264258 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.263910    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istiod-ca-cert" pod "c89aeca9-c4fc-4f9c-ba74-033b72349ddf" (UID: "c89aeca9-c4fc-4f9c-ba74-033b72349ddf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:10:32.264258 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.264129    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-credential-socket" pod "c89aeca9-c4fc-4f9c-ba74-033b72349ddf" (UID: "c89aeca9-c4fc-4f9c-ba74-033b72349ddf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:10:32.265700 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.264632    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:32.265700 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.264665    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:32.265700 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.264679    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:32.265700 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.264692    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:32.265700 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.264783    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-token" pod "c89aeca9-c4fc-4f9c-ba74-033b72349ddf" (UID: "c89aeca9-c4fc-4f9c-ba74-033b72349ddf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:10:32.265700 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.264943    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-kube-api-access-j4sfd" pod "c89aeca9-c4fc-4f9c-ba74-033b72349ddf" (UID: "c89aeca9-c4fc-4f9c-ba74-033b72349ddf"). InnerVolumeSpecName "kube-api-access-j4sfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:10:32.265700 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.265281    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-data" pod "c89aeca9-c4fc-4f9c-ba74-033b72349ddf" (UID: "c89aeca9-c4fc-4f9c-ba74-033b72349ddf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:10:32.265700 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.265408    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-envoy" pod "c89aeca9-c4fc-4f9c-ba74-033b72349ddf" (UID: "c89aeca9-c4fc-4f9c-ba74-033b72349ddf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:10:32.265700 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.265505    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-podinfo" pod "c89aeca9-c4fc-4f9c-ba74-033b72349ddf" (UID: "c89aeca9-c4fc-4f9c-ba74-033b72349ddf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:10:32.366093 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.366064    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4sfd\" (UniqueName: \"kubernetes.io/projected/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-kube-api-access-j4sfd\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:32.366093 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.366090    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:32.366093 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.366099    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:32.366368 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.366107    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:32.366368 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:32.366116    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c89aeca9-c4fc-4f9c-ba74-033b72349ddf-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:33.163994 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:33.163919    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p" event={"ID":"c89aeca9-c4fc-4f9c-ba74-033b72349ddf","Type":"ContainerDied","Data":"28da3b91fb96e593651e59a4bf5e0e3aebb6ce7a769226a52f64a82a537890de"}
Oct 30 05:10:33.163994 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:33.163976    4874 scope.go:122] "RemoveContainer" containerID="5efff0f058281139942ad6c2ab4fd98d67181a178e1b3d8cd72658c74a1475f5"
Oct 30 05:10:33.164459 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:33.163929    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"
Oct 30 05:10:33.227228 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:33.227190    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"]
Oct 30 05:10:33.237068 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:33.237009    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-7nr2p"]
Oct 30 05:10:34.731186 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:34.731151    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c89aeca9-c4fc-4f9c-ba74-033b72349ddf" path="/var/lib/kubelet/pods/c89aeca9-c4fc-4f9c-ba74-033b72349ddf/volumes"
Oct 30 05:10:35.961012 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:35.960971    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h"]
Oct 30 05:10:35.961473 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:35.961221    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h" podUID="b4c5e053-08b5-4351-acf8-2e57775a9366" containerName="maas-api" containerID="cri-o://2d4383550b518314b37a9c985062e4afa067cd8eca840f8d7b472be7f7b0f49d" gracePeriod=30
Oct 30 05:10:36.194063 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:36.193874    4874 generic.go:356] "Generic (PLEG): container finished" podID="b4c5e053-08b5-4351-acf8-2e57775a9366" containerID="2d4383550b518314b37a9c985062e4afa067cd8eca840f8d7b472be7f7b0f49d" exitCode=0
Oct 30 05:10:36.194063 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:36.193998    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h" event={"ID":"b4c5e053-08b5-4351-acf8-2e57775a9366","Type":"ContainerDied","Data":"2d4383550b518314b37a9c985062e4afa067cd8eca840f8d7b472be7f7b0f49d"}
Oct 30 05:10:36.204383 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:36.204352    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:10:36.358850 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:36.358482    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h"
Oct 30 05:10:36.440751 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:36.433432    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4c5e053-08b5-4351-acf8-2e57775a9366-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4c5e053-08b5-4351-acf8-2e57775a9366-openshift-service-ca\") pod \"b4c5e053-08b5-4351-acf8-2e57775a9366\" (UID: \"b4c5e053-08b5-4351-acf8-2e57775a9366\") "
Oct 30 05:10:36.440751 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:36.433706    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4c5e053-08b5-4351-acf8-2e57775a9366-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4c5e053-08b5-4351-acf8-2e57775a9366-maas-api-tls\") pod \"b4c5e053-08b5-4351-acf8-2e57775a9366\" (UID: \"b4c5e053-08b5-4351-acf8-2e57775a9366\") "
Oct 30 05:10:36.440751 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:36.433745    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4c5e053-08b5-4351-acf8-2e57775a9366-kube-api-access-ljvgk\" (UniqueName: \"kubernetes.io/projected/b4c5e053-08b5-4351-acf8-2e57775a9366-kube-api-access-ljvgk\") pod \"b4c5e053-08b5-4351-acf8-2e57775a9366\" (UID: \"b4c5e053-08b5-4351-acf8-2e57775a9366\") "
Oct 30 05:10:36.440751 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:36.433806    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4c5e053-08b5-4351-acf8-2e57775a9366-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4c5e053-08b5-4351-acf8-2e57775a9366-metrics-certs\") pod \"b4c5e053-08b5-4351-acf8-2e57775a9366\" (UID: \"b4c5e053-08b5-4351-acf8-2e57775a9366\") "
Oct 30 05:10:36.453261 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:36.444662    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4c5e053-08b5-4351-acf8-2e57775a9366-metrics-certs" pod "b4c5e053-08b5-4351-acf8-2e57775a9366" (UID: "b4c5e053-08b5-4351-acf8-2e57775a9366"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:10:36.466261 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:36.461424    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4c5e053-08b5-4351-acf8-2e57775a9366-openshift-service-ca" pod "b4c5e053-08b5-4351-acf8-2e57775a9366" (UID: "b4c5e053-08b5-4351-acf8-2e57775a9366"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:10:36.466261 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:36.462117    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c5e053-08b5-4351-acf8-2e57775a9366-kube-api-access-ljvgk" pod "b4c5e053-08b5-4351-acf8-2e57775a9366" (UID: "b4c5e053-08b5-4351-acf8-2e57775a9366"). InnerVolumeSpecName "kube-api-access-ljvgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:10:36.500438 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:36.500390    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4c5e053-08b5-4351-acf8-2e57775a9366-maas-api-tls" pod "b4c5e053-08b5-4351-acf8-2e57775a9366" (UID: "b4c5e053-08b5-4351-acf8-2e57775a9366"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:10:36.539229 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:36.539128    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4c5e053-08b5-4351-acf8-2e57775a9366-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:36.539229 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:36.539163    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4c5e053-08b5-4351-acf8-2e57775a9366-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:36.539229 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:36.539175    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4c5e053-08b5-4351-acf8-2e57775a9366-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:36.539229 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:36.539189    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljvgk\" (UniqueName: \"kubernetes.io/projected/b4c5e053-08b5-4351-acf8-2e57775a9366-kube-api-access-ljvgk\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:37.223468 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:37.223382    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h" event={"ID":"b4c5e053-08b5-4351-acf8-2e57775a9366","Type":"ContainerDied","Data":"f342d6e1a0bd825451c283054df36cc8466a8281937fa2fa30aa9cd3a514a5da"}
Oct 30 05:10:37.224033 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:37.224012    4874 scope.go:122] "RemoveContainer" containerID="2d4383550b518314b37a9c985062e4afa067cd8eca840f8d7b472be7f7b0f49d"
Oct 30 05:10:37.226054 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:37.225927    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h"
Oct 30 05:10:37.337803 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:37.337765    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h"]
Oct 30 05:10:37.368189 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:37.363061    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-66217b-7d46bdd6c7-mmk5h"]
Oct 30 05:10:38.736615 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:38.736574    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4c5e053-08b5-4351-acf8-2e57775a9366" path="/var/lib/kubelet/pods/b4c5e053-08b5-4351-acf8-2e57775a9366/volumes"
Oct 30 05:10:42.372936 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:42.372901    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"]
Oct 30 05:10:42.373485 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:42.373215    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6" podUID="17cf01ad-84ee-4f36-92f0-7cfff29f4215" containerName="istio-proxy" containerID="cri-o://cfd5317af1b4f6feba4f5881d93d8ee9f04554f05c57ce4a25ad7a6954d8f42f" gracePeriod=30
Oct 30 05:10:44.778251 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.778202    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"]
Oct 30 05:10:44.778785 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.778768    4874 cpu_manager.go:414] "containerMap: removing container" podUID="b4c5e053-08b5-4351-acf8-2e57775a9366" containerName="maas-api"
Oct 30 05:10:44.778785 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.778785    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4c5e053-08b5-4351-acf8-2e57775a9366" containerName="maas-api"
Oct 30 05:10:44.778921 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.778795    4874 cpu_manager.go:414] "containerMap: removing container" podUID="70bb7edd-3719-4c86-b358-2039dbd00675" containerName="maas-api"
Oct 30 05:10:44.778921 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.778800    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70bb7edd-3719-4c86-b358-2039dbd00675" containerName="maas-api"
Oct 30 05:10:44.778921 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.778817    4874 cpu_manager.go:414] "containerMap: removing container" podUID="c89aeca9-c4fc-4f9c-ba74-033b72349ddf" containerName="istio-proxy"
Oct 30 05:10:44.778921 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.778822    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c89aeca9-c4fc-4f9c-ba74-033b72349ddf" containerName="istio-proxy"
Oct 30 05:10:44.778921 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.778829    4874 cpu_manager.go:414] "containerMap: removing container" podUID="54b0d7fa-4b81-49a9-9819-2ee76bd5177e" containerName="istio-proxy"
Oct 30 05:10:44.778921 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.778833    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54b0d7fa-4b81-49a9-9819-2ee76bd5177e" containerName="istio-proxy"
Oct 30 05:10:44.778921 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.778860    4874 cpu_manager.go:414] "containerMap: removing container" podUID="2bea58a5-bdac-4545-9aee-66961d2bd86b" containerName="maas-api"
Oct 30 05:10:44.778921 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.778865    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bea58a5-bdac-4545-9aee-66961d2bd86b" containerName="maas-api"
Oct 30 05:10:44.779167 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.778948    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="c89aeca9-c4fc-4f9c-ba74-033b72349ddf" containerName="istio-proxy"
Oct 30 05:10:44.779167 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.778959    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="54b0d7fa-4b81-49a9-9819-2ee76bd5177e" containerName="istio-proxy"
Oct 30 05:10:44.779167 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.778969    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bea58a5-bdac-4545-9aee-66961d2bd86b" containerName="maas-api"
Oct 30 05:10:44.779167 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.778978    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4c5e053-08b5-4351-acf8-2e57775a9366" containerName="maas-api"
Oct 30 05:10:44.779167 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.778983    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="70bb7edd-3719-4c86-b358-2039dbd00675" containerName="maas-api"
Oct 30 05:10:44.782414 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.782398    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.796370 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.796346    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-247158-openshift-default-dockercfg-rvw8n\""
Oct 30 05:10:44.811728 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.811704    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"]
Oct 30 05:10:44.856876 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.856846    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-token\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.857003 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.856877    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-podinfo\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.857003 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.856899    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-workload-certs\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.857082 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.857036    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-data\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.857082 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.857063    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtwmz\" (UniqueName: \"kubernetes.io/projected/0d21e1c4-3882-438f-a391-b1697ff474cc-kube-api-access-gtwmz\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.857151 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.857113    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d21e1c4-3882-438f-a391-b1697ff474cc-istiod-ca-cert\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.857211 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.857190    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-credential-socket\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.857350 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.857331    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-envoy\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.857604 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.857581    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-workload-socket\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.958899 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.958867    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-data\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.958899 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.958901    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtwmz\" (UniqueName: \"kubernetes.io/projected/0d21e1c4-3882-438f-a391-b1697ff474cc-kube-api-access-gtwmz\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.959079 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.958921    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d21e1c4-3882-438f-a391-b1697ff474cc-istiod-ca-cert\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.959079 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.959043    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-credential-socket\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.959169 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.959152    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-envoy\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.959292 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.959275    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-workload-socket\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.959375 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.959328    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-data\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.959375 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.959363    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-token\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.959495 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.959393    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-podinfo\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.959495 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.959424    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-workload-certs\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.959495 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.959395    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-credential-socket\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.959644 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.959625    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-workload-socket\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.959702 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.959688    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-workload-certs\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.959752 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.959715    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d21e1c4-3882-438f-a391-b1697ff474cc-istiod-ca-cert\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.961402 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.961378    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-envoy\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.961794 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.961774    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-podinfo\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.981030 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.981010    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtwmz\" (UniqueName: \"kubernetes.io/projected/0d21e1c4-3882-438f-a391-b1697ff474cc-kube-api-access-gtwmz\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:44.981194 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:44.981172    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-token\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:45.096113 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:45.096025    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:45.420922 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:45.420896    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"]
Oct 30 05:10:45.422614 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:10:45.422587    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d21e1c4_3882_438f_a391_b1697ff474cc.slice/crio-4e44586e9d6633d5adb9e95caff5e13997930586e808ef6a22e08bcb6fa6cb08 WatchSource:0}: Error finding container 4e44586e9d6633d5adb9e95caff5e13997930586e808ef6a22e08bcb6fa6cb08: Status 404 returned error can't find the container with id 4e44586e9d6633d5adb9e95caff5e13997930586e808ef6a22e08bcb6fa6cb08
Oct 30 05:10:45.425060 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:45.425029    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 05:10:45.425152 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:45.425090    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 05:10:45.425152 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:45.425123    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 05:10:45.512194 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:45.512158    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:10:45.512376 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:45.512228    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6" podUID="17cf01ad-84ee-4f36-92f0-7cfff29f4215" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:10:46.279895 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:46.279856    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv" event={"ID":"0d21e1c4-3882-438f-a391-b1697ff474cc","Type":"ContainerStarted","Data":"ee0042b90997842b7585fff3531ba7fb5d63126f318a8b8cb431b6c11c0610ae"}
Oct 30 05:10:46.280314 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:46.279913    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv" event={"ID":"0d21e1c4-3882-438f-a391-b1697ff474cc","Type":"ContainerStarted","Data":"4e44586e9d6633d5adb9e95caff5e13997930586e808ef6a22e08bcb6fa6cb08"}
Oct 30 05:10:46.339228 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:46.339176    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv" podStartSLOduration=2.339160365 podStartE2EDuration="2.339160365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:10:46.335540576 +0000 UTC m=+4365.747936150" watchObservedRunningTime="2026-10-30 05:10:46.339160365 +0000 UTC m=+4365.751555937"
Oct 30 05:10:47.096548 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.096507    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:47.098368 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.098344    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 05:10:47.098436 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.098402    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv" podUID="0d21e1c4-3882-438f-a391-b1697ff474cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 05:10:47.710632 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.710608    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 05:10:47.816913 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.816880    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-workload-socket\") pod \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") "
Oct 30 05:10:47.817098 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.816979    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-podinfo\") pod \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") "
Oct 30 05:10:47.817098 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.817009    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-envoy\") pod \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") "
Oct 30 05:10:47.817098 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.817037    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-token\" (UniqueName: \"kubernetes.io/projected/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-token\") pod \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") "
Oct 30 05:10:47.817309 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.817118    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-data\") pod \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") "
Oct 30 05:10:47.817309 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.817192    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-workload-socket" pod "17cf01ad-84ee-4f36-92f0-7cfff29f4215" (UID: "17cf01ad-84ee-4f36-92f0-7cfff29f4215"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:10:47.817309 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.817226    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-credential-socket\") pod \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") "
Oct 30 05:10:47.817309 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.817300    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-workload-certs\") pod \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") "
Oct 30 05:10:47.817537 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.817413    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17cf01ad-84ee-4f36-92f0-7cfff29f4215-kube-api-access-frgxd\" (UniqueName: \"kubernetes.io/projected/17cf01ad-84ee-4f36-92f0-7cfff29f4215-kube-api-access-frgxd\") pod \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") "
Oct 30 05:10:47.817537 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.817431    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-data" pod "17cf01ad-84ee-4f36-92f0-7cfff29f4215" (UID: "17cf01ad-84ee-4f36-92f0-7cfff29f4215"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:10:47.817537 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.817453    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istiod-ca-cert\") pod \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\" (UID: \"17cf01ad-84ee-4f36-92f0-7cfff29f4215\") "
Oct 30 05:10:47.817716 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.817636    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-workload-certs" pod "17cf01ad-84ee-4f36-92f0-7cfff29f4215" (UID: "17cf01ad-84ee-4f36-92f0-7cfff29f4215"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:10:47.817915 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.817890    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-credential-socket" pod "17cf01ad-84ee-4f36-92f0-7cfff29f4215" (UID: "17cf01ad-84ee-4f36-92f0-7cfff29f4215"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:10:47.818012 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.817987    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istiod-ca-cert" pod "17cf01ad-84ee-4f36-92f0-7cfff29f4215" (UID: "17cf01ad-84ee-4f36-92f0-7cfff29f4215"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:10:47.818481 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.818452    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:47.818607 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.818484    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:47.818607 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.818499    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:47.818607 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.818513    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:47.818607 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.818526    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:47.820107 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.820084    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-podinfo" pod "17cf01ad-84ee-4f36-92f0-7cfff29f4215" (UID: "17cf01ad-84ee-4f36-92f0-7cfff29f4215"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:10:47.820259 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.820124    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-token" pod "17cf01ad-84ee-4f36-92f0-7cfff29f4215" (UID: "17cf01ad-84ee-4f36-92f0-7cfff29f4215"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:10:47.820643 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.820427    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17cf01ad-84ee-4f36-92f0-7cfff29f4215-kube-api-access-frgxd" pod "17cf01ad-84ee-4f36-92f0-7cfff29f4215" (UID: "17cf01ad-84ee-4f36-92f0-7cfff29f4215"). InnerVolumeSpecName "kube-api-access-frgxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:10:47.821226 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.821193    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-envoy" pod "17cf01ad-84ee-4f36-92f0-7cfff29f4215" (UID: "17cf01ad-84ee-4f36-92f0-7cfff29f4215"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:10:47.852084 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.852057    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"]
Oct 30 05:10:47.920298 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.920207    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frgxd\" (UniqueName: \"kubernetes.io/projected/17cf01ad-84ee-4f36-92f0-7cfff29f4215-kube-api-access-frgxd\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:47.920298 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.920260    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:47.920298 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.920271    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:47.920298 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:47.920279    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17cf01ad-84ee-4f36-92f0-7cfff29f4215-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:10:48.096413 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:48.096378    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 05:10:48.096595 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:48.096428    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv" podUID="0d21e1c4-3882-438f-a391-b1697ff474cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 05:10:48.304328 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:48.304291    4874 generic.go:356] "Generic (PLEG): container finished" podID="17cf01ad-84ee-4f36-92f0-7cfff29f4215" containerID="cfd5317af1b4f6feba4f5881d93d8ee9f04554f05c57ce4a25ad7a6954d8f42f" exitCode=0
Oct 30 05:10:48.304500 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:48.304354    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"
Oct 30 05:10:48.304500 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:48.304379    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6" event={"ID":"17cf01ad-84ee-4f36-92f0-7cfff29f4215","Type":"ContainerDied","Data":"cfd5317af1b4f6feba4f5881d93d8ee9f04554f05c57ce4a25ad7a6954d8f42f"}
Oct 30 05:10:48.304500 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:48.304441    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6" event={"ID":"17cf01ad-84ee-4f36-92f0-7cfff29f4215","Type":"ContainerDied","Data":"2638a119f6ec44d73eb4bfb79e1fe17b8dcd5cbf38641e7f1fe0057f058e9e4c"}
Oct 30 05:10:48.304500 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:48.304468    4874 scope.go:122] "RemoveContainer" containerID="cfd5317af1b4f6feba4f5881d93d8ee9f04554f05c57ce4a25ad7a6954d8f42f"
Oct 30 05:10:48.315170 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:48.315153    4874 scope.go:122] "RemoveContainer" containerID="b7b8d7e33783e6f609daf6a5979ea50cbe9f93708e0d823482a02b39d91ac504"
Oct 30 05:10:48.324065 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:48.324042    4874 scope.go:122] "RemoveContainer" containerID="cfd5317af1b4f6feba4f5881d93d8ee9f04554f05c57ce4a25ad7a6954d8f42f"
Oct 30 05:10:48.324328 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:10:48.324309    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfd5317af1b4f6feba4f5881d93d8ee9f04554f05c57ce4a25ad7a6954d8f42f\": container with ID starting with cfd5317af1b4f6feba4f5881d93d8ee9f04554f05c57ce4a25ad7a6954d8f42f not found: ID does not exist" containerID="cfd5317af1b4f6feba4f5881d93d8ee9f04554f05c57ce4a25ad7a6954d8f42f"
Oct 30 05:10:48.324459 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:48.324335    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd5317af1b4f6feba4f5881d93d8ee9f04554f05c57ce4a25ad7a6954d8f42f"} err="failed to get container status \"cfd5317af1b4f6feba4f5881d93d8ee9f04554f05c57ce4a25ad7a6954d8f42f\": rpc error: code = NotFound desc = could not find container \"cfd5317af1b4f6feba4f5881d93d8ee9f04554f05c57ce4a25ad7a6954d8f42f\": container with ID starting with cfd5317af1b4f6feba4f5881d93d8ee9f04554f05c57ce4a25ad7a6954d8f42f not found: ID does not exist"
Oct 30 05:10:48.324459 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:48.324350    4874 scope.go:122] "RemoveContainer" containerID="b7b8d7e33783e6f609daf6a5979ea50cbe9f93708e0d823482a02b39d91ac504"
Oct 30 05:10:48.324577 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:10:48.324563    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7b8d7e33783e6f609daf6a5979ea50cbe9f93708e0d823482a02b39d91ac504\": container with ID starting with b7b8d7e33783e6f609daf6a5979ea50cbe9f93708e0d823482a02b39d91ac504 not found: ID does not exist" containerID="b7b8d7e33783e6f609daf6a5979ea50cbe9f93708e0d823482a02b39d91ac504"
Oct 30 05:10:48.324636 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:48.324582    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7b8d7e33783e6f609daf6a5979ea50cbe9f93708e0d823482a02b39d91ac504"} err="failed to get container status \"b7b8d7e33783e6f609daf6a5979ea50cbe9f93708e0d823482a02b39d91ac504\": rpc error: code = NotFound desc = could not find container \"b7b8d7e33783e6f609daf6a5979ea50cbe9f93708e0d823482a02b39d91ac504\": container with ID starting with b7b8d7e33783e6f609daf6a5979ea50cbe9f93708e0d823482a02b39d91ac504 not found: ID does not exist"
Oct 30 05:10:48.355039 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:48.355014    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"]
Oct 30 05:10:48.361427 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:48.361406    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-66217b-openshift-default-7666bc75bf-9j6d6"]
Oct 30 05:10:48.729309 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:48.729281    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17cf01ad-84ee-4f36-92f0-7cfff29f4215" path="/var/lib/kubelet/pods/17cf01ad-84ee-4f36-92f0-7cfff29f4215/volumes"
Oct 30 05:10:49.106363 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:49.104618    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 05:10:49.106363 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:49.104695    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv" podUID="0d21e1c4-3882-438f-a391-b1697ff474cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 05:10:50.103263 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:50.102606    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 05:10:50.103263 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:50.102665    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv" podUID="0d21e1c4-3882-438f-a391-b1697ff474cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 05:10:51.097381 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:51.097344    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 05:10:51.097578 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:51.097399    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv" podUID="0d21e1c4-3882-438f-a391-b1697ff474cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 05:10:52.096944 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:52.096904    4874 patch_prober.go:29] interesting pod/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 05:10:52.097385 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:52.096970    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv" podUID="0d21e1c4-3882-438f-a391-b1697ff474cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 05:10:53.101827 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:53.101796    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:53.102253 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:53.102069    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:53.102723 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:53.102706    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:10:55.608533 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:10:55.608497    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:10:56.377159 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:56.377121    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh"]
Oct 30 05:10:56.415950 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:10:56.415913    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7"]
Oct 30 05:11:01.388421 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:01.388347    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh" podUID="e1774f86-5e25-4358-b250-d26c6ea6c601" containerName="payload-processing" containerID="cri-o://84b66fd27a4a954920fa49279432373a78222a97d62a58de6e3cc8d9fa019334" gracePeriod=25
Oct 30 05:11:01.428275 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:01.424321    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7" podUID="5b386201-0f5e-4add-98a8-bb3bcf476089" containerName="payload-pre-processing" containerID="cri-o://a5fe5dc9d0a8e870cbec1fdba0fb588052c0309b0dfc2a6cdb3de66754e64cf6" gracePeriod=25
Oct 30 05:11:01.541859 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:01.541817    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh" podUID="e1774f86-5e25-4358-b250-d26c6ea6c601" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:11:02.559151 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:02.559113    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7" podUID="5b386201-0f5e-4add-98a8-bb3bcf476089" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:11:02.897156 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:02.897070    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7" podUID="1463c013-e615-4142-b799-56b17e67a474" containerName="main" containerID="cri-o://339f649f7b6920cc8a9bb5a59c418959bff8afe37a63d9bb9f4d0e5aa5ee52a6" gracePeriod=45
Oct 30 05:11:03.011183 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.011142    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7" podUID="1463c013-e615-4142-b799-56b17e67a474" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": dial tcp 10.134.0.51:8000: connect: connection refused"
Oct 30 05:11:03.154738 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.154681    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 05:11:03.264267 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.264203    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-dshm\") pod \"1463c013-e615-4142-b799-56b17e67a474\" (UID: \"1463c013-e615-4142-b799-56b17e67a474\") "
Oct 30 05:11:03.264460 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.264296    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1463c013-e615-4142-b799-56b17e67a474-kube-api-access-nc75q\" (UniqueName: \"kubernetes.io/projected/1463c013-e615-4142-b799-56b17e67a474-kube-api-access-nc75q\") pod \"1463c013-e615-4142-b799-56b17e67a474\" (UID: \"1463c013-e615-4142-b799-56b17e67a474\") "
Oct 30 05:11:03.264460 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.264386    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-tmp-dir\") pod \"1463c013-e615-4142-b799-56b17e67a474\" (UID: \"1463c013-e615-4142-b799-56b17e67a474\") "
Oct 30 05:11:03.264460 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.264414    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-model-cache\") pod \"1463c013-e615-4142-b799-56b17e67a474\" (UID: \"1463c013-e615-4142-b799-56b17e67a474\") "
Oct 30 05:11:03.264648 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.264544    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-home\" (UniqueName: \"kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-home\") pod \"1463c013-e615-4142-b799-56b17e67a474\" (UID: \"1463c013-e615-4142-b799-56b17e67a474\") "
Oct 30 05:11:03.264721 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.264686    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1463c013-e615-4142-b799-56b17e67a474-tls-certs\" (UniqueName: \"kubernetes.io/secret/1463c013-e615-4142-b799-56b17e67a474-tls-certs\") pod \"1463c013-e615-4142-b799-56b17e67a474\" (UID: \"1463c013-e615-4142-b799-56b17e67a474\") "
Oct 30 05:11:03.264721 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.264685    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-tmp-dir" pod "1463c013-e615-4142-b799-56b17e67a474" (UID: "1463c013-e615-4142-b799-56b17e67a474"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:03.264832 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.264768    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-home" pod "1463c013-e615-4142-b799-56b17e67a474" (UID: "1463c013-e615-4142-b799-56b17e67a474"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:03.264926 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.264906    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-model-cache" pod "1463c013-e615-4142-b799-56b17e67a474" (UID: "1463c013-e615-4142-b799-56b17e67a474"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:03.265205 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.265184    4874 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-tmp-dir\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:03.265334 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.265206    4874 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-model-cache\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:03.265334 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.265219    4874 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-home\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:03.266561 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.266535    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-dshm" pod "1463c013-e615-4142-b799-56b17e67a474" (UID: "1463c013-e615-4142-b799-56b17e67a474"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:03.266783 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.266764    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1463c013-e615-4142-b799-56b17e67a474-kube-api-access-nc75q" pod "1463c013-e615-4142-b799-56b17e67a474" (UID: "1463c013-e615-4142-b799-56b17e67a474"). InnerVolumeSpecName "kube-api-access-nc75q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:11:03.266847 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.266826    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1463c013-e615-4142-b799-56b17e67a474-tls-certs" pod "1463c013-e615-4142-b799-56b17e67a474" (UID: "1463c013-e615-4142-b799-56b17e67a474"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:11:03.366761 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.366731    4874 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1463c013-e615-4142-b799-56b17e67a474-tls-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:03.366761 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.366759    4874 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1463c013-e615-4142-b799-56b17e67a474-dshm\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:03.366987 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.366769    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc75q\" (UniqueName: \"kubernetes.io/projected/1463c013-e615-4142-b799-56b17e67a474-kube-api-access-nc75q\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:03.449489 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.449406    4874 generic.go:356] "Generic (PLEG): container finished" podID="1463c013-e615-4142-b799-56b17e67a474" containerID="339f649f7b6920cc8a9bb5a59c418959bff8afe37a63d9bb9f4d0e5aa5ee52a6" exitCode=0
Oct 30 05:11:03.449489 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.449450    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"
Oct 30 05:11:03.449705 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.449491    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7" event={"ID":"1463c013-e615-4142-b799-56b17e67a474","Type":"ContainerDied","Data":"339f649f7b6920cc8a9bb5a59c418959bff8afe37a63d9bb9f4d0e5aa5ee52a6"}
Oct 30 05:11:03.449705 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.449533    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7" event={"ID":"1463c013-e615-4142-b799-56b17e67a474","Type":"ContainerDied","Data":"a335dd9a92d28a27f1fb0a184d4d9785b9602e3d2e7b6cd41b20d1354ee54921"}
Oct 30 05:11:03.449705 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.449549    4874 scope.go:122] "RemoveContainer" containerID="339f649f7b6920cc8a9bb5a59c418959bff8afe37a63d9bb9f4d0e5aa5ee52a6"
Oct 30 05:11:03.460076 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.460061    4874 scope.go:122] "RemoveContainer" containerID="339f649f7b6920cc8a9bb5a59c418959bff8afe37a63d9bb9f4d0e5aa5ee52a6"
Oct 30 05:11:03.460343 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:11:03.460324    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"339f649f7b6920cc8a9bb5a59c418959bff8afe37a63d9bb9f4d0e5aa5ee52a6\": container with ID starting with 339f649f7b6920cc8a9bb5a59c418959bff8afe37a63d9bb9f4d0e5aa5ee52a6 not found: ID does not exist" containerID="339f649f7b6920cc8a9bb5a59c418959bff8afe37a63d9bb9f4d0e5aa5ee52a6"
Oct 30 05:11:03.460388 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.460349    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339f649f7b6920cc8a9bb5a59c418959bff8afe37a63d9bb9f4d0e5aa5ee52a6"} err="failed to get container status \"339f649f7b6920cc8a9bb5a59c418959bff8afe37a63d9bb9f4d0e5aa5ee52a6\": rpc error: code = NotFound desc = could not find container \"339f649f7b6920cc8a9bb5a59c418959bff8afe37a63d9bb9f4d0e5aa5ee52a6\": container with ID starting with 339f649f7b6920cc8a9bb5a59c418959bff8afe37a63d9bb9f4d0e5aa5ee52a6 not found: ID does not exist"
Oct 30 05:11:03.513717 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.513681    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"]
Oct 30 05:11:03.518135 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:03.518110    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-5d74b9/premium-w2-5d74b9-kserve-76956ddc5c-pdqw7"]
Oct 30 05:11:04.730632 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:04.730599    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1463c013-e615-4142-b799-56b17e67a474" path="/var/lib/kubelet/pods/1463c013-e615-4142-b799-56b17e67a474/volumes"
Oct 30 05:11:06.641684 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:06.641650    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:11:07.931397 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:07.931368    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"]
Oct 30 05:11:08.010273 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:07.980203    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"]
Oct 30 05:11:11.541594 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:11.541554    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh" podUID="e1774f86-5e25-4358-b250-d26c6ea6c601" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:11:12.550162 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:12.550123    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7" podUID="5b386201-0f5e-4add-98a8-bb3bcf476089" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:11:14.515371 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:14.515337    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4"]
Oct 30 05:11:14.515852 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:14.515598    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4" podUID="ea64a87b-85b1-46fa-9c51-4e45543bf808" containerName="maas-api" containerID="cri-o://495dbb28cd4ac9b43fe51c3e9b2062aff29ac3ddb654c314fc0d00b0124765c6" gracePeriod=30
Oct 30 05:11:14.842732 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:14.842650    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4"
Oct 30 05:11:15.023443 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:15.023409    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea64a87b-85b1-46fa-9c51-4e45543bf808-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea64a87b-85b1-46fa-9c51-4e45543bf808-metrics-certs\") pod \"ea64a87b-85b1-46fa-9c51-4e45543bf808\" (UID: \"ea64a87b-85b1-46fa-9c51-4e45543bf808\") "
Oct 30 05:11:15.024153 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:15.024133    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea64a87b-85b1-46fa-9c51-4e45543bf808-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea64a87b-85b1-46fa-9c51-4e45543bf808-maas-api-tls\") pod \"ea64a87b-85b1-46fa-9c51-4e45543bf808\" (UID: \"ea64a87b-85b1-46fa-9c51-4e45543bf808\") "
Oct 30 05:11:15.024401 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:15.024382    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea64a87b-85b1-46fa-9c51-4e45543bf808-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea64a87b-85b1-46fa-9c51-4e45543bf808-openshift-service-ca\") pod \"ea64a87b-85b1-46fa-9c51-4e45543bf808\" (UID: \"ea64a87b-85b1-46fa-9c51-4e45543bf808\") "
Oct 30 05:11:15.024648 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:15.024629    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea64a87b-85b1-46fa-9c51-4e45543bf808-kube-api-access-6kmmd\" (UniqueName: \"kubernetes.io/projected/ea64a87b-85b1-46fa-9c51-4e45543bf808-kube-api-access-6kmmd\") pod \"ea64a87b-85b1-46fa-9c51-4e45543bf808\" (UID: \"ea64a87b-85b1-46fa-9c51-4e45543bf808\") "
Oct 30 05:11:15.024901 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:15.024873    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea64a87b-85b1-46fa-9c51-4e45543bf808-openshift-service-ca" pod "ea64a87b-85b1-46fa-9c51-4e45543bf808" (UID: "ea64a87b-85b1-46fa-9c51-4e45543bf808"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:11:15.025396 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:15.025374    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea64a87b-85b1-46fa-9c51-4e45543bf808-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:15.026275 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:15.026253    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea64a87b-85b1-46fa-9c51-4e45543bf808-metrics-certs" pod "ea64a87b-85b1-46fa-9c51-4e45543bf808" (UID: "ea64a87b-85b1-46fa-9c51-4e45543bf808"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:11:15.029663 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:15.029637    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea64a87b-85b1-46fa-9c51-4e45543bf808-maas-api-tls" pod "ea64a87b-85b1-46fa-9c51-4e45543bf808" (UID: "ea64a87b-85b1-46fa-9c51-4e45543bf808"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:11:15.036313 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:15.036286    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea64a87b-85b1-46fa-9c51-4e45543bf808-kube-api-access-6kmmd" pod "ea64a87b-85b1-46fa-9c51-4e45543bf808" (UID: "ea64a87b-85b1-46fa-9c51-4e45543bf808"). InnerVolumeSpecName "kube-api-access-6kmmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:11:15.126974 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:15.126908    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea64a87b-85b1-46fa-9c51-4e45543bf808-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:15.126974 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:15.126932    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea64a87b-85b1-46fa-9c51-4e45543bf808-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:15.126974 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:15.126941    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kmmd\" (UniqueName: \"kubernetes.io/projected/ea64a87b-85b1-46fa-9c51-4e45543bf808-kube-api-access-6kmmd\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:15.529519 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:15.529479    4874 generic.go:356] "Generic (PLEG): container finished" podID="ea64a87b-85b1-46fa-9c51-4e45543bf808" containerID="495dbb28cd4ac9b43fe51c3e9b2062aff29ac3ddb654c314fc0d00b0124765c6" exitCode=0
Oct 30 05:11:15.529994 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:15.529538    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4"
Oct 30 05:11:15.529994 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:15.529559    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4" event={"ID":"ea64a87b-85b1-46fa-9c51-4e45543bf808","Type":"ContainerDied","Data":"495dbb28cd4ac9b43fe51c3e9b2062aff29ac3ddb654c314fc0d00b0124765c6"}
Oct 30 05:11:15.529994 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:15.529647    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4" event={"ID":"ea64a87b-85b1-46fa-9c51-4e45543bf808","Type":"ContainerDied","Data":"f4d98b94f1feee06b6ac9f579a2c08e04be9cbfc45fc13580f79d4cd318cc92b"}
Oct 30 05:11:15.529994 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:15.529683    4874 scope.go:122] "RemoveContainer" containerID="495dbb28cd4ac9b43fe51c3e9b2062aff29ac3ddb654c314fc0d00b0124765c6"
Oct 30 05:11:15.541002 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:15.540981    4874 scope.go:122] "RemoveContainer" containerID="495dbb28cd4ac9b43fe51c3e9b2062aff29ac3ddb654c314fc0d00b0124765c6"
Oct 30 05:11:15.541291 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:11:15.541266    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"495dbb28cd4ac9b43fe51c3e9b2062aff29ac3ddb654c314fc0d00b0124765c6\": container with ID starting with 495dbb28cd4ac9b43fe51c3e9b2062aff29ac3ddb654c314fc0d00b0124765c6 not found: ID does not exist" containerID="495dbb28cd4ac9b43fe51c3e9b2062aff29ac3ddb654c314fc0d00b0124765c6"
Oct 30 05:11:15.541371 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:15.541299    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495dbb28cd4ac9b43fe51c3e9b2062aff29ac3ddb654c314fc0d00b0124765c6"} err="failed to get container status \"495dbb28cd4ac9b43fe51c3e9b2062aff29ac3ddb654c314fc0d00b0124765c6\": rpc error: code = NotFound desc = could not find container \"495dbb28cd4ac9b43fe51c3e9b2062aff29ac3ddb654c314fc0d00b0124765c6\": container with ID starting with 495dbb28cd4ac9b43fe51c3e9b2062aff29ac3ddb654c314fc0d00b0124765c6 not found: ID does not exist"
Oct 30 05:11:15.583296 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:15.583260    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4"]
Oct 30 05:11:15.592702 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:15.592673    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-5d74b9-6b9dcb9c74-bbwg4"]
Oct 30 05:11:16.730019 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:16.729989    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea64a87b-85b1-46fa-9c51-4e45543bf808" path="/var/lib/kubelet/pods/ea64a87b-85b1-46fa-9c51-4e45543bf808/volumes"
Oct 30 05:11:21.541675 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:11:21.541642    4874 logging.go:55] [core] [Channel #6760 SubChannel #6762]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.43:50052", ServerName: "10.134.0.43:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.43:50052: connect: connection refused"
Oct 30 05:11:21.567570 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:21.567538    4874 generic.go:356] "Generic (PLEG): container finished" podID="5b386201-0f5e-4add-98a8-bb3bcf476089" containerID="a5fe5dc9d0a8e870cbec1fdba0fb588052c0309b0dfc2a6cdb3de66754e64cf6" exitCode=0
Oct 30 05:11:21.567754 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:21.567736    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7" event={"ID":"5b386201-0f5e-4add-98a8-bb3bcf476089","Type":"ContainerDied","Data":"a5fe5dc9d0a8e870cbec1fdba0fb588052c0309b0dfc2a6cdb3de66754e64cf6"}
Oct 30 05:11:21.569588 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:21.569565    4874 generic.go:356] "Generic (PLEG): container finished" podID="e1774f86-5e25-4358-b250-d26c6ea6c601" containerID="84b66fd27a4a954920fa49279432373a78222a97d62a58de6e3cc8d9fa019334" exitCode=0
Oct 30 05:11:21.569714 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:21.569608    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh" event={"ID":"e1774f86-5e25-4358-b250-d26c6ea6c601","Type":"ContainerDied","Data":"84b66fd27a4a954920fa49279432373a78222a97d62a58de6e3cc8d9fa019334"}
Oct 30 05:11:21.698399 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:21.698373    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh"
Oct 30 05:11:21.721616 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:21.721585    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7"
Oct 30 05:11:21.860718 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:21.860686    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1774f86-5e25-4358-b250-d26c6ea6c601-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1774f86-5e25-4358-b250-d26c6ea6c601-plugins-config-volume\") pod \"e1774f86-5e25-4358-b250-d26c6ea6c601\" (UID: \"e1774f86-5e25-4358-b250-d26c6ea6c601\") "
Oct 30 05:11:21.860907 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:21.860731    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b386201-0f5e-4add-98a8-bb3bcf476089-kube-api-access-c4lgz\" (UniqueName: \"kubernetes.io/projected/5b386201-0f5e-4add-98a8-bb3bcf476089-kube-api-access-c4lgz\") pod \"5b386201-0f5e-4add-98a8-bb3bcf476089\" (UID: \"5b386201-0f5e-4add-98a8-bb3bcf476089\") "
Oct 30 05:11:21.860907 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:21.860829    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1774f86-5e25-4358-b250-d26c6ea6c601-kube-api-access-7vbh4\" (UniqueName: \"kubernetes.io/projected/e1774f86-5e25-4358-b250-d26c6ea6c601-kube-api-access-7vbh4\") pod \"e1774f86-5e25-4358-b250-d26c6ea6c601\" (UID: \"e1774f86-5e25-4358-b250-d26c6ea6c601\") "
Oct 30 05:11:21.861037 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:21.860915    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b386201-0f5e-4add-98a8-bb3bcf476089-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b386201-0f5e-4add-98a8-bb3bcf476089-plugins-config-volume\") pod \"5b386201-0f5e-4add-98a8-bb3bcf476089\" (UID: \"5b386201-0f5e-4add-98a8-bb3bcf476089\") "
Oct 30 05:11:21.861203 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:21.861172    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1774f86-5e25-4358-b250-d26c6ea6c601-plugins-config-volume" pod "e1774f86-5e25-4358-b250-d26c6ea6c601" (UID: "e1774f86-5e25-4358-b250-d26c6ea6c601"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:11:21.861365 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:21.861293    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b386201-0f5e-4add-98a8-bb3bcf476089-plugins-config-volume" pod "5b386201-0f5e-4add-98a8-bb3bcf476089" (UID: "5b386201-0f5e-4add-98a8-bb3bcf476089"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:11:21.861519 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:21.861500    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b386201-0f5e-4add-98a8-bb3bcf476089-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:21.861598 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:21.861523    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1774f86-5e25-4358-b250-d26c6ea6c601-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:21.863042 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:21.863021    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b386201-0f5e-4add-98a8-bb3bcf476089-kube-api-access-c4lgz" pod "5b386201-0f5e-4add-98a8-bb3bcf476089" (UID: "5b386201-0f5e-4add-98a8-bb3bcf476089"). InnerVolumeSpecName "kube-api-access-c4lgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:11:21.863136 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:21.863039    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1774f86-5e25-4358-b250-d26c6ea6c601-kube-api-access-7vbh4" pod "e1774f86-5e25-4358-b250-d26c6ea6c601" (UID: "e1774f86-5e25-4358-b250-d26c6ea6c601"). InnerVolumeSpecName "kube-api-access-7vbh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:11:21.962853 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:21.962788    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vbh4\" (UniqueName: \"kubernetes.io/projected/e1774f86-5e25-4358-b250-d26c6ea6c601-kube-api-access-7vbh4\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:21.962853 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:21.962813    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4lgz\" (UniqueName: \"kubernetes.io/projected/5b386201-0f5e-4add-98a8-bb3bcf476089-kube-api-access-c4lgz\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:22.543265 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:22.541419    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh" podUID="e1774f86-5e25-4358-b250-d26c6ea6c601" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.43:50052\" within 1s: context deadline exceeded"
Oct 30 05:11:22.543265 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:22.541528    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh"
Oct 30 05:11:22.577452 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:22.577425    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh"
Oct 30 05:11:22.577623 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:22.577472    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh" event={"ID":"e1774f86-5e25-4358-b250-d26c6ea6c601","Type":"ContainerDied","Data":"b4212c87e6403b78797fc3bb48bc0f90ec2c30e1467a6fae587c32943bc6ba14"}
Oct 30 05:11:22.577623 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:22.577536    4874 scope.go:122] "RemoveContainer" containerID="84b66fd27a4a954920fa49279432373a78222a97d62a58de6e3cc8d9fa019334"
Oct 30 05:11:22.578954 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:22.578912    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7"
Oct 30 05:11:22.578954 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:22.578943    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7" event={"ID":"5b386201-0f5e-4add-98a8-bb3bcf476089","Type":"ContainerDied","Data":"0d6a8888e43b048b4fa2ff7068c549bc39ff94b1185c610485e85f811b369478"}
Oct 30 05:11:22.589572 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:22.589551    4874 scope.go:122] "RemoveContainer" containerID="a5fe5dc9d0a8e870cbec1fdba0fb588052c0309b0dfc2a6cdb3de66754e64cf6"
Oct 30 05:11:22.626695 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:22.626670    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7"]
Oct 30 05:11:22.637420 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:22.637400    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5d74b9-58f7b8cdc9-5hvv7"]
Oct 30 05:11:22.669355 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:22.669298    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh"]
Oct 30 05:11:22.673745 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:22.673726    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5d74b9-58b6754b68-mb7rh"]
Oct 30 05:11:22.729628 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:22.729601    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b386201-0f5e-4add-98a8-bb3bcf476089" path="/var/lib/kubelet/pods/5b386201-0f5e-4add-98a8-bb3bcf476089/volumes"
Oct 30 05:11:22.730019 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:22.730006    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1774f86-5e25-4358-b250-d26c6ea6c601" path="/var/lib/kubelet/pods/e1774f86-5e25-4358-b250-d26c6ea6c601/volumes"
Oct 30 05:11:23.179366 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.179332    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k" podUID="0e28b1d5-cbaa-40ea-b74c-fecb28d84a38" containerName="main" containerID="cri-o://73fad9de186bacea3765e08dc4af3d10a41eccb24fcffb8acd8a6baca0d87f15" gracePeriod=45
Oct 30 05:11:23.191655 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.191621    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx" podUID="70ea964b-efdb-4d99-b4f5-e3cde4331254" containerName="main" containerID="cri-o://475b08288960784cda9d585c8d84e3ed5fb19d7f6a246f17942db07e87850e06" gracePeriod=45
Oct 30 05:11:23.292560 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.292524    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx" podUID="70ea964b-efdb-4d99-b4f5-e3cde4331254" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": dial tcp 10.134.0.53:8000: connect: connection refused"
Oct 30 05:11:23.496658 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.496635    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 05:11:23.499613 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.499588    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 05:11:23.582625 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.581638    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-kube-api-access-mqjsc\" (UniqueName: \"kubernetes.io/projected/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-kube-api-access-mqjsc\") pod \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\" (UID: \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\") "
Oct 30 05:11:23.582625 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.581710    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-tmp-dir\") pod \"70ea964b-efdb-4d99-b4f5-e3cde4331254\" (UID: \"70ea964b-efdb-4d99-b4f5-e3cde4331254\") "
Oct 30 05:11:23.582625 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.581762    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-tls-certs\" (UniqueName: \"kubernetes.io/secret/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-tls-certs\") pod \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\" (UID: \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\") "
Oct 30 05:11:23.582625 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.581830    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-home\" (UniqueName: \"kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-home\") pod \"70ea964b-efdb-4d99-b4f5-e3cde4331254\" (UID: \"70ea964b-efdb-4d99-b4f5-e3cde4331254\") "
Oct 30 05:11:23.582625 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.581882    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-dshm\") pod \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\" (UID: \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\") "
Oct 30 05:11:23.582625 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.581945    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70ea964b-efdb-4d99-b4f5-e3cde4331254-kube-api-access-pfltx\" (UniqueName: \"kubernetes.io/projected/70ea964b-efdb-4d99-b4f5-e3cde4331254-kube-api-access-pfltx\") pod \"70ea964b-efdb-4d99-b4f5-e3cde4331254\" (UID: \"70ea964b-efdb-4d99-b4f5-e3cde4331254\") "
Oct 30 05:11:23.582625 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.581972    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-home\" (UniqueName: \"kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-home\") pod \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\" (UID: \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\") "
Oct 30 05:11:23.582625 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.582012    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-model-cache\") pod \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\" (UID: \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\") "
Oct 30 05:11:23.582625 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.582044    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-tmp-dir\") pod \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\" (UID: \"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38\") "
Oct 30 05:11:23.582625 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.582110    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-dshm\") pod \"70ea964b-efdb-4d99-b4f5-e3cde4331254\" (UID: \"70ea964b-efdb-4d99-b4f5-e3cde4331254\") "
Oct 30 05:11:23.582625 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.582130    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-home" pod "70ea964b-efdb-4d99-b4f5-e3cde4331254" (UID: "70ea964b-efdb-4d99-b4f5-e3cde4331254"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:23.582625 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.582168    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70ea964b-efdb-4d99-b4f5-e3cde4331254-tls-certs\" (UniqueName: \"kubernetes.io/secret/70ea964b-efdb-4d99-b4f5-e3cde4331254-tls-certs\") pod \"70ea964b-efdb-4d99-b4f5-e3cde4331254\" (UID: \"70ea964b-efdb-4d99-b4f5-e3cde4331254\") "
Oct 30 05:11:23.582625 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.582225    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-model-cache\") pod \"70ea964b-efdb-4d99-b4f5-e3cde4331254\" (UID: \"70ea964b-efdb-4d99-b4f5-e3cde4331254\") "
Oct 30 05:11:23.582625 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.582453    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-tmp-dir" pod "70ea964b-efdb-4d99-b4f5-e3cde4331254" (UID: "70ea964b-efdb-4d99-b4f5-e3cde4331254"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:23.584035 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.582910    4874 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-tmp-dir\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:23.584035 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.582934    4874 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-home\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:23.584035 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.583140    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-tmp-dir" pod "0e28b1d5-cbaa-40ea-b74c-fecb28d84a38" (UID: "0e28b1d5-cbaa-40ea-b74c-fecb28d84a38"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:23.584035 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.583335    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-home" pod "0e28b1d5-cbaa-40ea-b74c-fecb28d84a38" (UID: "0e28b1d5-cbaa-40ea-b74c-fecb28d84a38"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:23.584035 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.583462    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-model-cache" pod "0e28b1d5-cbaa-40ea-b74c-fecb28d84a38" (UID: "0e28b1d5-cbaa-40ea-b74c-fecb28d84a38"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:23.584741 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.584424    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-model-cache" pod "70ea964b-efdb-4d99-b4f5-e3cde4331254" (UID: "70ea964b-efdb-4d99-b4f5-e3cde4331254"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:23.585828 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.585802    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70ea964b-efdb-4d99-b4f5-e3cde4331254-tls-certs" pod "70ea964b-efdb-4d99-b4f5-e3cde4331254" (UID: "70ea964b-efdb-4d99-b4f5-e3cde4331254"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:11:23.588924 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.586961    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-dshm" pod "0e28b1d5-cbaa-40ea-b74c-fecb28d84a38" (UID: "0e28b1d5-cbaa-40ea-b74c-fecb28d84a38"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:23.589265 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.589043    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-kube-api-access-mqjsc" pod "0e28b1d5-cbaa-40ea-b74c-fecb28d84a38" (UID: "0e28b1d5-cbaa-40ea-b74c-fecb28d84a38"). InnerVolumeSpecName "kube-api-access-mqjsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:11:23.589787 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.589754    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-dshm" pod "70ea964b-efdb-4d99-b4f5-e3cde4331254" (UID: "70ea964b-efdb-4d99-b4f5-e3cde4331254"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:11:23.590176 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.590148    4874 generic.go:356] "Generic (PLEG): container finished" podID="0e28b1d5-cbaa-40ea-b74c-fecb28d84a38" containerID="73fad9de186bacea3765e08dc4af3d10a41eccb24fcffb8acd8a6baca0d87f15" exitCode=0
Oct 30 05:11:23.590342 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.590269    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k" event={"ID":"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38","Type":"ContainerDied","Data":"73fad9de186bacea3765e08dc4af3d10a41eccb24fcffb8acd8a6baca0d87f15"}
Oct 30 05:11:23.590342 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.590312    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k" event={"ID":"0e28b1d5-cbaa-40ea-b74c-fecb28d84a38","Type":"ContainerDied","Data":"05221d6414bf902a631e5bd81d7d60e4a9f7e44153a13f403fd30c140a69e20f"}
Oct 30 05:11:23.590342 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.590333    4874 scope.go:122] "RemoveContainer" containerID="73fad9de186bacea3765e08dc4af3d10a41eccb24fcffb8acd8a6baca0d87f15"
Oct 30 05:11:23.590498 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.590406    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"
Oct 30 05:11:23.590622 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.590599    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ea964b-efdb-4d99-b4f5-e3cde4331254-kube-api-access-pfltx" pod "70ea964b-efdb-4d99-b4f5-e3cde4331254" (UID: "70ea964b-efdb-4d99-b4f5-e3cde4331254"). InnerVolumeSpecName "kube-api-access-pfltx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:11:23.590743 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.590635    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-tls-certs" pod "0e28b1d5-cbaa-40ea-b74c-fecb28d84a38" (UID: "0e28b1d5-cbaa-40ea-b74c-fecb28d84a38"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:11:23.593433 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.593410    4874 generic.go:356] "Generic (PLEG): container finished" podID="70ea964b-efdb-4d99-b4f5-e3cde4331254" containerID="475b08288960784cda9d585c8d84e3ed5fb19d7f6a246f17942db07e87850e06" exitCode=0
Oct 30 05:11:23.593607 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.593460    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"
Oct 30 05:11:23.593607 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.593475    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx" event={"ID":"70ea964b-efdb-4d99-b4f5-e3cde4331254","Type":"ContainerDied","Data":"475b08288960784cda9d585c8d84e3ed5fb19d7f6a246f17942db07e87850e06"}
Oct 30 05:11:23.593607 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.593508    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx" event={"ID":"70ea964b-efdb-4d99-b4f5-e3cde4331254","Type":"ContainerDied","Data":"01a9abdd1d1c671018271e21649a5ee6c58f3f04348fa6f893ca3cb745c8e4f6"}
Oct 30 05:11:23.610799 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.610783    4874 scope.go:122] "RemoveContainer" containerID="73fad9de186bacea3765e08dc4af3d10a41eccb24fcffb8acd8a6baca0d87f15"
Oct 30 05:11:23.611107 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:11:23.611086    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73fad9de186bacea3765e08dc4af3d10a41eccb24fcffb8acd8a6baca0d87f15\": container with ID starting with 73fad9de186bacea3765e08dc4af3d10a41eccb24fcffb8acd8a6baca0d87f15 not found: ID does not exist" containerID="73fad9de186bacea3765e08dc4af3d10a41eccb24fcffb8acd8a6baca0d87f15"
Oct 30 05:11:23.611188 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.611118    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73fad9de186bacea3765e08dc4af3d10a41eccb24fcffb8acd8a6baca0d87f15"} err="failed to get container status \"73fad9de186bacea3765e08dc4af3d10a41eccb24fcffb8acd8a6baca0d87f15\": rpc error: code = NotFound desc = could not find container \"73fad9de186bacea3765e08dc4af3d10a41eccb24fcffb8acd8a6baca0d87f15\": container with ID starting with 73fad9de186bacea3765e08dc4af3d10a41eccb24fcffb8acd8a6baca0d87f15 not found: ID does not exist"
Oct 30 05:11:23.611188 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.611140    4874 scope.go:122] "RemoveContainer" containerID="475b08288960784cda9d585c8d84e3ed5fb19d7f6a246f17942db07e87850e06"
Oct 30 05:11:23.621122 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.621076    4874 scope.go:122] "RemoveContainer" containerID="475b08288960784cda9d585c8d84e3ed5fb19d7f6a246f17942db07e87850e06"
Oct 30 05:11:23.621398 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:11:23.621380    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"475b08288960784cda9d585c8d84e3ed5fb19d7f6a246f17942db07e87850e06\": container with ID starting with 475b08288960784cda9d585c8d84e3ed5fb19d7f6a246f17942db07e87850e06 not found: ID does not exist" containerID="475b08288960784cda9d585c8d84e3ed5fb19d7f6a246f17942db07e87850e06"
Oct 30 05:11:23.621598 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.621403    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"475b08288960784cda9d585c8d84e3ed5fb19d7f6a246f17942db07e87850e06"} err="failed to get container status \"475b08288960784cda9d585c8d84e3ed5fb19d7f6a246f17942db07e87850e06\": rpc error: code = NotFound desc = could not find container \"475b08288960784cda9d585c8d84e3ed5fb19d7f6a246f17942db07e87850e06\": container with ID starting with 475b08288960784cda9d585c8d84e3ed5fb19d7f6a246f17942db07e87850e06 not found: ID does not exist"
Oct 30 05:11:23.666096 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.665294    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"]
Oct 30 05:11:23.666096 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.665343    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/premium-w1-6796b1-kserve-578b7d867d-glskx"]
Oct 30 05:11:23.683902 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.683863    4874 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-dshm\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:23.684027 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.683918    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfltx\" (UniqueName: \"kubernetes.io/projected/70ea964b-efdb-4d99-b4f5-e3cde4331254-kube-api-access-pfltx\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:23.684027 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.683929    4874 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-home\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:23.684027 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.683938    4874 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-model-cache\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:23.684027 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.683947    4874 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-tmp-dir\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:23.684027 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.683954    4874 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-dshm\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:23.684027 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.683961    4874 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ea964b-efdb-4d99-b4f5-e3cde4331254-tls-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:23.684027 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.683969    4874 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ea964b-efdb-4d99-b4f5-e3cde4331254-model-cache\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:23.684027 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.683976    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqjsc\" (UniqueName: \"kubernetes.io/projected/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-kube-api-access-mqjsc\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:23.684027 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.683983    4874 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38-tls-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:11:23.935487 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.935455    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"]
Oct 30 05:11:23.937880 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:23.937858    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/distinct-w1-6796b1-kserve-5bcd5d98cc-jzl6k"]
Oct 30 05:11:24.766260 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:24.764793    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e28b1d5-cbaa-40ea-b74c-fecb28d84a38" path="/var/lib/kubelet/pods/0e28b1d5-cbaa-40ea-b74c-fecb28d84a38/volumes"
Oct 30 05:11:24.766260 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:24.765573    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70ea964b-efdb-4d99-b4f5-e3cde4331254" path="/var/lib/kubelet/pods/70ea964b-efdb-4d99-b4f5-e3cde4331254/volumes"
Oct 30 05:11:25.608880 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:11:25.608843    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:11:55.610075 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:11:55.610038    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:11:59.863517 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:11:59.863477    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:12:02.106121 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:02.106089    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb"]
Oct 30 05:12:02.106549 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:02.106388    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb" podUID="8d278356-d7d2-42cd-bba4-e5ef1867fa1c" containerName="maas-api" containerID="cri-o://6e9c4196c6c3c1a630fe2804f2dafdc1beae2a555157227f4d74b968f9a0fa40" gracePeriod=30
Oct 30 05:12:02.383207 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:02.383123    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb"
Oct 30 05:12:02.443718 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:02.442510    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-metrics-certs\") pod \"8d278356-d7d2-42cd-bba4-e5ef1867fa1c\" (UID: \"8d278356-d7d2-42cd-bba4-e5ef1867fa1c\") "
Oct 30 05:12:02.443718 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:02.442569    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-openshift-service-ca\") pod \"8d278356-d7d2-42cd-bba4-e5ef1867fa1c\" (UID: \"8d278356-d7d2-42cd-bba4-e5ef1867fa1c\") "
Oct 30 05:12:02.443718 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:02.442652    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-maas-api-tls\") pod \"8d278356-d7d2-42cd-bba4-e5ef1867fa1c\" (UID: \"8d278356-d7d2-42cd-bba4-e5ef1867fa1c\") "
Oct 30 05:12:02.443718 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:02.442797    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-kube-api-access-vhkdd\" (UniqueName: \"kubernetes.io/projected/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-kube-api-access-vhkdd\") pod \"8d278356-d7d2-42cd-bba4-e5ef1867fa1c\" (UID: \"8d278356-d7d2-42cd-bba4-e5ef1867fa1c\") "
Oct 30 05:12:02.445147 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:02.445099    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-openshift-service-ca" pod "8d278356-d7d2-42cd-bba4-e5ef1867fa1c" (UID: "8d278356-d7d2-42cd-bba4-e5ef1867fa1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:12:02.445941 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:02.445898    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-kube-api-access-vhkdd" pod "8d278356-d7d2-42cd-bba4-e5ef1867fa1c" (UID: "8d278356-d7d2-42cd-bba4-e5ef1867fa1c"). InnerVolumeSpecName "kube-api-access-vhkdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:12:02.448117 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:02.448092    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-metrics-certs" pod "8d278356-d7d2-42cd-bba4-e5ef1867fa1c" (UID: "8d278356-d7d2-42cd-bba4-e5ef1867fa1c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:12:02.450537 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:02.450516    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-maas-api-tls" pod "8d278356-d7d2-42cd-bba4-e5ef1867fa1c" (UID: "8d278356-d7d2-42cd-bba4-e5ef1867fa1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:12:02.544704 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:02.544666    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:12:02.544704 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:02.544706    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:12:02.544704 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:02.544721    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhkdd\" (UniqueName: \"kubernetes.io/projected/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-kube-api-access-vhkdd\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:12:02.545089 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:02.544734    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d278356-d7d2-42cd-bba4-e5ef1867fa1c-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:12:02.909620 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:02.909584    4874 generic.go:356] "Generic (PLEG): container finished" podID="8d278356-d7d2-42cd-bba4-e5ef1867fa1c" containerID="6e9c4196c6c3c1a630fe2804f2dafdc1beae2a555157227f4d74b968f9a0fa40" exitCode=0
Oct 30 05:12:02.909806 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:02.909631    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb"
Oct 30 05:12:02.909806 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:02.909675    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb" event={"ID":"8d278356-d7d2-42cd-bba4-e5ef1867fa1c","Type":"ContainerDied","Data":"6e9c4196c6c3c1a630fe2804f2dafdc1beae2a555157227f4d74b968f9a0fa40"}
Oct 30 05:12:02.909806 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:02.909726    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb" event={"ID":"8d278356-d7d2-42cd-bba4-e5ef1867fa1c","Type":"ContainerDied","Data":"8609e8f0ec7b2a6b97198453485a3bef71d272d1cb5272cbd40dc6f41f74d1c1"}
Oct 30 05:12:02.909806 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:02.909756    4874 scope.go:122] "RemoveContainer" containerID="6e9c4196c6c3c1a630fe2804f2dafdc1beae2a555157227f4d74b968f9a0fa40"
Oct 30 05:12:02.919066 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:02.919045    4874 scope.go:122] "RemoveContainer" containerID="6e9c4196c6c3c1a630fe2804f2dafdc1beae2a555157227f4d74b968f9a0fa40"
Oct 30 05:12:02.919380 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:12:02.919360    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e9c4196c6c3c1a630fe2804f2dafdc1beae2a555157227f4d74b968f9a0fa40\": container with ID starting with 6e9c4196c6c3c1a630fe2804f2dafdc1beae2a555157227f4d74b968f9a0fa40 not found: ID does not exist" containerID="6e9c4196c6c3c1a630fe2804f2dafdc1beae2a555157227f4d74b968f9a0fa40"
Oct 30 05:12:02.919474 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:02.919386    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e9c4196c6c3c1a630fe2804f2dafdc1beae2a555157227f4d74b968f9a0fa40"} err="failed to get container status \"6e9c4196c6c3c1a630fe2804f2dafdc1beae2a555157227f4d74b968f9a0fa40\": rpc error: code = NotFound desc = could not find container \"6e9c4196c6c3c1a630fe2804f2dafdc1beae2a555157227f4d74b968f9a0fa40\": container with ID starting with 6e9c4196c6c3c1a630fe2804f2dafdc1beae2a555157227f4d74b968f9a0fa40 not found: ID does not exist"
Oct 30 05:12:02.958897 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:02.958869    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb"]
Oct 30 05:12:02.963669 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:02.963644    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-6796b1-74b57d4859-4nwhb"]
Oct 30 05:12:04.729620 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:04.729587    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d278356-d7d2-42cd-bba4-e5ef1867fa1c" path="/var/lib/kubelet/pods/8d278356-d7d2-42cd-bba4-e5ef1867fa1c/volumes"
Oct 30 05:12:05.995178 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:05.995145    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5"]
Oct 30 05:12:05.995660 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:05.995408    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5" podUID="a849651f-2575-40e3-9f53-3c2b057075ce" containerName="maas-api" containerID="cri-o://62d8b40844774686e5bc85c02bbc6be065e7c2a07d34cace68443a968d78f653" gracePeriod=30
Oct 30 05:12:06.170780 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:06.169277    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff"]
Oct 30 05:12:06.280511 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:06.280212    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5"
Oct 30 05:12:06.402489 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:06.402453    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a849651f-2575-40e3-9f53-3c2b057075ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a849651f-2575-40e3-9f53-3c2b057075ce-metrics-certs\") pod \"a849651f-2575-40e3-9f53-3c2b057075ce\" (UID: \"a849651f-2575-40e3-9f53-3c2b057075ce\") "
Oct 30 05:12:06.402673 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:06.402550    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a849651f-2575-40e3-9f53-3c2b057075ce-kube-api-access-qxzfp\" (UniqueName: \"kubernetes.io/projected/a849651f-2575-40e3-9f53-3c2b057075ce-kube-api-access-qxzfp\") pod \"a849651f-2575-40e3-9f53-3c2b057075ce\" (UID: \"a849651f-2575-40e3-9f53-3c2b057075ce\") "
Oct 30 05:12:06.402741 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:06.402712    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a849651f-2575-40e3-9f53-3c2b057075ce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a849651f-2575-40e3-9f53-3c2b057075ce-openshift-service-ca\") pod \"a849651f-2575-40e3-9f53-3c2b057075ce\" (UID: \"a849651f-2575-40e3-9f53-3c2b057075ce\") "
Oct 30 05:12:06.402807 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:06.402746    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a849651f-2575-40e3-9f53-3c2b057075ce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a849651f-2575-40e3-9f53-3c2b057075ce-maas-api-tls\") pod \"a849651f-2575-40e3-9f53-3c2b057075ce\" (UID: \"a849651f-2575-40e3-9f53-3c2b057075ce\") "
Oct 30 05:12:06.403112 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:06.403079    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a849651f-2575-40e3-9f53-3c2b057075ce-openshift-service-ca" pod "a849651f-2575-40e3-9f53-3c2b057075ce" (UID: "a849651f-2575-40e3-9f53-3c2b057075ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:12:06.403342 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:06.403316    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a849651f-2575-40e3-9f53-3c2b057075ce-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:12:06.405586 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:06.405547    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a849651f-2575-40e3-9f53-3c2b057075ce-metrics-certs" pod "a849651f-2575-40e3-9f53-3c2b057075ce" (UID: "a849651f-2575-40e3-9f53-3c2b057075ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:12:06.405586 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:06.405565    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a849651f-2575-40e3-9f53-3c2b057075ce-maas-api-tls" pod "a849651f-2575-40e3-9f53-3c2b057075ce" (UID: "a849651f-2575-40e3-9f53-3c2b057075ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:12:06.405986 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:06.405963    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a849651f-2575-40e3-9f53-3c2b057075ce-kube-api-access-qxzfp" pod "a849651f-2575-40e3-9f53-3c2b057075ce" (UID: "a849651f-2575-40e3-9f53-3c2b057075ce"). InnerVolumeSpecName "kube-api-access-qxzfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:12:06.504882 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:06.504816    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxzfp\" (UniqueName: \"kubernetes.io/projected/a849651f-2575-40e3-9f53-3c2b057075ce-kube-api-access-qxzfp\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:12:06.504882 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:06.504844    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a849651f-2575-40e3-9f53-3c2b057075ce-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:12:06.504882 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:06.504854    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a849651f-2575-40e3-9f53-3c2b057075ce-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:12:06.961082 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:06.960986    4874 generic.go:356] "Generic (PLEG): container finished" podID="a849651f-2575-40e3-9f53-3c2b057075ce" containerID="62d8b40844774686e5bc85c02bbc6be065e7c2a07d34cace68443a968d78f653" exitCode=0
Oct 30 05:12:06.961287 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:06.961152    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5" event={"ID":"a849651f-2575-40e3-9f53-3c2b057075ce","Type":"ContainerDied","Data":"62d8b40844774686e5bc85c02bbc6be065e7c2a07d34cace68443a968d78f653"}
Oct 30 05:12:06.961287 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:06.961202    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5" event={"ID":"a849651f-2575-40e3-9f53-3c2b057075ce","Type":"ContainerDied","Data":"13f94d77ff4c302c19867b3b3e87c3ba0ecfb44ea44ab89b4ee1e380f95136a0"}
Oct 30 05:12:06.961287 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:06.961234    4874 scope.go:122] "RemoveContainer" containerID="62d8b40844774686e5bc85c02bbc6be065e7c2a07d34cace68443a968d78f653"
Oct 30 05:12:06.961507 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:06.961406    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5"
Oct 30 05:12:06.972962 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:06.972932    4874 scope.go:122] "RemoveContainer" containerID="62d8b40844774686e5bc85c02bbc6be065e7c2a07d34cace68443a968d78f653"
Oct 30 05:12:06.973275 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:12:06.973256    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62d8b40844774686e5bc85c02bbc6be065e7c2a07d34cace68443a968d78f653\": container with ID starting with 62d8b40844774686e5bc85c02bbc6be065e7c2a07d34cace68443a968d78f653 not found: ID does not exist" containerID="62d8b40844774686e5bc85c02bbc6be065e7c2a07d34cace68443a968d78f653"
Oct 30 05:12:06.973347 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:06.973285    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d8b40844774686e5bc85c02bbc6be065e7c2a07d34cace68443a968d78f653"} err="failed to get container status \"62d8b40844774686e5bc85c02bbc6be065e7c2a07d34cace68443a968d78f653\": rpc error: code = NotFound desc = could not find container \"62d8b40844774686e5bc85c02bbc6be065e7c2a07d34cace68443a968d78f653\": container with ID starting with 62d8b40844774686e5bc85c02bbc6be065e7c2a07d34cace68443a968d78f653 not found: ID does not exist"
Oct 30 05:12:07.010343 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:07.010318    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5"]
Oct 30 05:12:07.021440 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:07.021414    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0d6e7-85447cc547-zx6m5"]
Oct 30 05:12:08.730482 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:08.730453    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a849651f-2575-40e3-9f53-3c2b057075ce" path="/var/lib/kubelet/pods/a849651f-2575-40e3-9f53-3c2b057075ce/volumes"
Oct 30 05:12:11.170624 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:11.170561    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff" podUID="2ac6eaed-819d-4ce2-b299-b7228312da81" containerName="payload-pre-processing" containerID="cri-o://e00621fb17f97b000e874824cb41d4b196741af3e00781a6213922ffe02a8532" gracePeriod=25
Oct 30 05:12:11.424734 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:11.424667    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff"
Oct 30 05:12:11.567019 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:11.566990    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ac6eaed-819d-4ce2-b299-b7228312da81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ac6eaed-819d-4ce2-b299-b7228312da81-plugins-config-volume\") pod \"2ac6eaed-819d-4ce2-b299-b7228312da81\" (UID: \"2ac6eaed-819d-4ce2-b299-b7228312da81\") "
Oct 30 05:12:11.567176 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:11.567095    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ac6eaed-819d-4ce2-b299-b7228312da81-kube-api-access-57bf4\" (UniqueName: \"kubernetes.io/projected/2ac6eaed-819d-4ce2-b299-b7228312da81-kube-api-access-57bf4\") pod \"2ac6eaed-819d-4ce2-b299-b7228312da81\" (UID: \"2ac6eaed-819d-4ce2-b299-b7228312da81\") "
Oct 30 05:12:11.567447 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:11.567423    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac6eaed-819d-4ce2-b299-b7228312da81-plugins-config-volume" pod "2ac6eaed-819d-4ce2-b299-b7228312da81" (UID: "2ac6eaed-819d-4ce2-b299-b7228312da81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:12:11.569137 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:11.569120    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac6eaed-819d-4ce2-b299-b7228312da81-kube-api-access-57bf4" pod "2ac6eaed-819d-4ce2-b299-b7228312da81" (UID: "2ac6eaed-819d-4ce2-b299-b7228312da81"). InnerVolumeSpecName "kube-api-access-57bf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:12:11.679313 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:11.676074    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ac6eaed-819d-4ce2-b299-b7228312da81-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:12:11.679313 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:11.676112    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57bf4\" (UniqueName: \"kubernetes.io/projected/2ac6eaed-819d-4ce2-b299-b7228312da81-kube-api-access-57bf4\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:12:11.991589 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:11.991555    4874 generic.go:356] "Generic (PLEG): container finished" podID="2ac6eaed-819d-4ce2-b299-b7228312da81" containerID="e00621fb17f97b000e874824cb41d4b196741af3e00781a6213922ffe02a8532" exitCode=0
Oct 30 05:12:11.991766 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:11.991600    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff"
Oct 30 05:12:11.991766 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:11.991635    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff" event={"ID":"2ac6eaed-819d-4ce2-b299-b7228312da81","Type":"ContainerDied","Data":"e00621fb17f97b000e874824cb41d4b196741af3e00781a6213922ffe02a8532"}
Oct 30 05:12:11.991766 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:11.991691    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff" event={"ID":"2ac6eaed-819d-4ce2-b299-b7228312da81","Type":"ContainerDied","Data":"942f0bf3cf95d381bded240bb5c5ec7b59b2333444886ec742132983b8a4db4e"}
Oct 30 05:12:11.991766 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:11.991718    4874 scope.go:122] "RemoveContainer" containerID="e00621fb17f97b000e874824cb41d4b196741af3e00781a6213922ffe02a8532"
Oct 30 05:12:12.002135 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:12.002115    4874 scope.go:122] "RemoveContainer" containerID="e00621fb17f97b000e874824cb41d4b196741af3e00781a6213922ffe02a8532"
Oct 30 05:12:12.002514 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:12:12.002496    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e00621fb17f97b000e874824cb41d4b196741af3e00781a6213922ffe02a8532\": container with ID starting with e00621fb17f97b000e874824cb41d4b196741af3e00781a6213922ffe02a8532 not found: ID does not exist" containerID="e00621fb17f97b000e874824cb41d4b196741af3e00781a6213922ffe02a8532"
Oct 30 05:12:12.002574 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:12.002523    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e00621fb17f97b000e874824cb41d4b196741af3e00781a6213922ffe02a8532"} err="failed to get container status \"e00621fb17f97b000e874824cb41d4b196741af3e00781a6213922ffe02a8532\": rpc error: code = NotFound desc = could not find container \"e00621fb17f97b000e874824cb41d4b196741af3e00781a6213922ffe02a8532\": container with ID starting with e00621fb17f97b000e874824cb41d4b196741af3e00781a6213922ffe02a8532 not found: ID does not exist"
Oct 30 05:12:12.040632 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:12.040609    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff"]
Oct 30 05:12:12.051581 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:12.051559    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c0d6e7-859b8d8c47-v55ff"]
Oct 30 05:12:12.730292 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:12.730257    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ac6eaed-819d-4ce2-b299-b7228312da81" path="/var/lib/kubelet/pods/2ac6eaed-819d-4ce2-b299-b7228312da81/volumes"
Oct 30 05:12:21.996568 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:21.996489    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274"]
Oct 30 05:12:22.029766 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:22.029734    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv"]
Oct 30 05:12:25.608808 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:12:25.608776    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:12:26.996840 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:26.996778    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274" podUID="b9c8bf07-6e78-4a49-a32c-302fe418d7cc" containerName="payload-processing" containerID="cri-o://036aba7c3b1892cda2ef41553fac984cc9b009805807ab79c9a8777407b18fbf" gracePeriod=25
Oct 30 05:12:27.030586 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:27.030547    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv" podUID="b71a43c4-1c1e-4e92-9a3d-68b6827efa7b" containerName="payload-pre-processing" containerID="cri-o://06053bfc780a89e82ba93a8cd7ff5a512721e4dba54a62cf376cd1365c543a5d" gracePeriod=25
Oct 30 05:12:27.310035 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:27.310010    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274"
Oct 30 05:12:27.316416 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:27.316396    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv"
Oct 30 05:12:27.480923 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:27.480893    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9c8bf07-6e78-4a49-a32c-302fe418d7cc-kube-api-access-m8xq5\" (UniqueName: \"kubernetes.io/projected/b9c8bf07-6e78-4a49-a32c-302fe418d7cc-kube-api-access-m8xq5\") pod \"b9c8bf07-6e78-4a49-a32c-302fe418d7cc\" (UID: \"b9c8bf07-6e78-4a49-a32c-302fe418d7cc\") "
Oct 30 05:12:27.481130 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:27.481059    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9c8bf07-6e78-4a49-a32c-302fe418d7cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c8bf07-6e78-4a49-a32c-302fe418d7cc-plugins-config-volume\") pod \"b9c8bf07-6e78-4a49-a32c-302fe418d7cc\" (UID: \"b9c8bf07-6e78-4a49-a32c-302fe418d7cc\") "
Oct 30 05:12:27.481130 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:27.481098    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b71a43c4-1c1e-4e92-9a3d-68b6827efa7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b71a43c4-1c1e-4e92-9a3d-68b6827efa7b-plugins-config-volume\") pod \"b71a43c4-1c1e-4e92-9a3d-68b6827efa7b\" (UID: \"b71a43c4-1c1e-4e92-9a3d-68b6827efa7b\") "
Oct 30 05:12:27.481277 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:27.481161    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b71a43c4-1c1e-4e92-9a3d-68b6827efa7b-kube-api-access-j7qgw\" (UniqueName: \"kubernetes.io/projected/b71a43c4-1c1e-4e92-9a3d-68b6827efa7b-kube-api-access-j7qgw\") pod \"b71a43c4-1c1e-4e92-9a3d-68b6827efa7b\" (UID: \"b71a43c4-1c1e-4e92-9a3d-68b6827efa7b\") "
Oct 30 05:12:27.481502 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:27.481472    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c8bf07-6e78-4a49-a32c-302fe418d7cc-plugins-config-volume" pod "b9c8bf07-6e78-4a49-a32c-302fe418d7cc" (UID: "b9c8bf07-6e78-4a49-a32c-302fe418d7cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:12:27.481502 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:27.481491    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71a43c4-1c1e-4e92-9a3d-68b6827efa7b-plugins-config-volume" pod "b71a43c4-1c1e-4e92-9a3d-68b6827efa7b" (UID: "b71a43c4-1c1e-4e92-9a3d-68b6827efa7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:12:27.481706 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:27.481584    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c8bf07-6e78-4a49-a32c-302fe418d7cc-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:12:27.481706 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:27.481600    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b71a43c4-1c1e-4e92-9a3d-68b6827efa7b-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:12:27.483010 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:27.482990    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c8bf07-6e78-4a49-a32c-302fe418d7cc-kube-api-access-m8xq5" pod "b9c8bf07-6e78-4a49-a32c-302fe418d7cc" (UID: "b9c8bf07-6e78-4a49-a32c-302fe418d7cc"). InnerVolumeSpecName "kube-api-access-m8xq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:12:27.483147 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:27.483128    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b71a43c4-1c1e-4e92-9a3d-68b6827efa7b-kube-api-access-j7qgw" pod "b71a43c4-1c1e-4e92-9a3d-68b6827efa7b" (UID: "b71a43c4-1c1e-4e92-9a3d-68b6827efa7b"). InnerVolumeSpecName "kube-api-access-j7qgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:12:27.582661 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:27.582588    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8xq5\" (UniqueName: \"kubernetes.io/projected/b9c8bf07-6e78-4a49-a32c-302fe418d7cc-kube-api-access-m8xq5\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:12:27.582661 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:27.582615    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7qgw\" (UniqueName: \"kubernetes.io/projected/b71a43c4-1c1e-4e92-9a3d-68b6827efa7b-kube-api-access-j7qgw\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:12:28.087176 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:28.087144    4874 generic.go:356] "Generic (PLEG): container finished" podID="b9c8bf07-6e78-4a49-a32c-302fe418d7cc" containerID="036aba7c3b1892cda2ef41553fac984cc9b009805807ab79c9a8777407b18fbf" exitCode=0
Oct 30 05:12:28.087658 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:28.087194    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274"
Oct 30 05:12:28.087658 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:28.087264    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274" event={"ID":"b9c8bf07-6e78-4a49-a32c-302fe418d7cc","Type":"ContainerDied","Data":"036aba7c3b1892cda2ef41553fac984cc9b009805807ab79c9a8777407b18fbf"}
Oct 30 05:12:28.087658 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:28.087318    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274" event={"ID":"b9c8bf07-6e78-4a49-a32c-302fe418d7cc","Type":"ContainerDied","Data":"c424a7c22263f644389026694d8d890d344a98ccf565fcbbd1feb72af9764414"}
Oct 30 05:12:28.087658 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:28.087342    4874 scope.go:122] "RemoveContainer" containerID="036aba7c3b1892cda2ef41553fac984cc9b009805807ab79c9a8777407b18fbf"
Oct 30 05:12:28.088754 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:28.088733    4874 generic.go:356] "Generic (PLEG): container finished" podID="b71a43c4-1c1e-4e92-9a3d-68b6827efa7b" containerID="06053bfc780a89e82ba93a8cd7ff5a512721e4dba54a62cf376cd1365c543a5d" exitCode=0
Oct 30 05:12:28.088879 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:28.088773    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv" event={"ID":"b71a43c4-1c1e-4e92-9a3d-68b6827efa7b","Type":"ContainerDied","Data":"06053bfc780a89e82ba93a8cd7ff5a512721e4dba54a62cf376cd1365c543a5d"}
Oct 30 05:12:28.088879 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:28.088784    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv"
Oct 30 05:12:28.088879 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:28.088810    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv" event={"ID":"b71a43c4-1c1e-4e92-9a3d-68b6827efa7b","Type":"ContainerDied","Data":"a4b4cf22b50e26b19761473b1af742e4671a5afdbcbd964c0d4db5c21f6b37dc"}
Oct 30 05:12:28.098049 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:28.098029    4874 scope.go:122] "RemoveContainer" containerID="036aba7c3b1892cda2ef41553fac984cc9b009805807ab79c9a8777407b18fbf"
Oct 30 05:12:28.098436 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:12:28.098411    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"036aba7c3b1892cda2ef41553fac984cc9b009805807ab79c9a8777407b18fbf\": container with ID starting with 036aba7c3b1892cda2ef41553fac984cc9b009805807ab79c9a8777407b18fbf not found: ID does not exist" containerID="036aba7c3b1892cda2ef41553fac984cc9b009805807ab79c9a8777407b18fbf"
Oct 30 05:12:28.098522 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:28.098444    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"036aba7c3b1892cda2ef41553fac984cc9b009805807ab79c9a8777407b18fbf"} err="failed to get container status \"036aba7c3b1892cda2ef41553fac984cc9b009805807ab79c9a8777407b18fbf\": rpc error: code = NotFound desc = could not find container \"036aba7c3b1892cda2ef41553fac984cc9b009805807ab79c9a8777407b18fbf\": container with ID starting with 036aba7c3b1892cda2ef41553fac984cc9b009805807ab79c9a8777407b18fbf not found: ID does not exist"
Oct 30 05:12:28.098522 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:28.098462    4874 scope.go:122] "RemoveContainer" containerID="06053bfc780a89e82ba93a8cd7ff5a512721e4dba54a62cf376cd1365c543a5d"
Oct 30 05:12:28.107154 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:28.107139    4874 scope.go:122] "RemoveContainer" containerID="06053bfc780a89e82ba93a8cd7ff5a512721e4dba54a62cf376cd1365c543a5d"
Oct 30 05:12:28.107382 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:12:28.107363    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06053bfc780a89e82ba93a8cd7ff5a512721e4dba54a62cf376cd1365c543a5d\": container with ID starting with 06053bfc780a89e82ba93a8cd7ff5a512721e4dba54a62cf376cd1365c543a5d not found: ID does not exist" containerID="06053bfc780a89e82ba93a8cd7ff5a512721e4dba54a62cf376cd1365c543a5d"
Oct 30 05:12:28.107448 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:28.107389    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06053bfc780a89e82ba93a8cd7ff5a512721e4dba54a62cf376cd1365c543a5d"} err="failed to get container status \"06053bfc780a89e82ba93a8cd7ff5a512721e4dba54a62cf376cd1365c543a5d\": rpc error: code = NotFound desc = could not find container \"06053bfc780a89e82ba93a8cd7ff5a512721e4dba54a62cf376cd1365c543a5d\": container with ID starting with 06053bfc780a89e82ba93a8cd7ff5a512721e4dba54a62cf376cd1365c543a5d not found: ID does not exist"
Oct 30 05:12:28.135114 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:28.135089    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274"]
Oct 30 05:12:28.146495 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:28.146475    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fcf42a-7b6b696d58-9q274"]
Oct 30 05:12:28.180126 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:28.180092    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv"]
Oct 30 05:12:28.186347 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:28.186321    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fcf42a-7f456f8d55-vqwgv"]
Oct 30 05:12:28.730391 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:28.730363    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b71a43c4-1c1e-4e92-9a3d-68b6827efa7b" path="/var/lib/kubelet/pods/b71a43c4-1c1e-4e92-9a3d-68b6827efa7b/volumes"
Oct 30 05:12:28.730763 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:28.730750    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9c8bf07-6e78-4a49-a32c-302fe418d7cc" path="/var/lib/kubelet/pods/b9c8bf07-6e78-4a49-a32c-302fe418d7cc/volumes"
Oct 30 05:12:33.255451 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:33.255417    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw"]
Oct 30 05:12:33.255899 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:33.255644    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw" podUID="9b7e4826-cc1f-401c-ad81-608f4234cf7f" containerName="maas-api" containerID="cri-o://4d607ea1e08474103072523084939bc732dc5767ede2a3f8e9c89f0334b709c6" gracePeriod=30
Oct 30 05:12:33.603626 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:33.603599    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw"
Oct 30 05:12:33.675642 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:33.675599    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b7e4826-cc1f-401c-ad81-608f4234cf7f-kube-api-access-kc4v5\" (UniqueName: \"kubernetes.io/projected/9b7e4826-cc1f-401c-ad81-608f4234cf7f-kube-api-access-kc4v5\") pod \"9b7e4826-cc1f-401c-ad81-608f4234cf7f\" (UID: \"9b7e4826-cc1f-401c-ad81-608f4234cf7f\") "
Oct 30 05:12:33.675906 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:33.675885    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b7e4826-cc1f-401c-ad81-608f4234cf7f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7e4826-cc1f-401c-ad81-608f4234cf7f-openshift-service-ca\") pod \"9b7e4826-cc1f-401c-ad81-608f4234cf7f\" (UID: \"9b7e4826-cc1f-401c-ad81-608f4234cf7f\") "
Oct 30 05:12:33.676034 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:33.676019    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b7e4826-cc1f-401c-ad81-608f4234cf7f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b7e4826-cc1f-401c-ad81-608f4234cf7f-metrics-certs\") pod \"9b7e4826-cc1f-401c-ad81-608f4234cf7f\" (UID: \"9b7e4826-cc1f-401c-ad81-608f4234cf7f\") "
Oct 30 05:12:33.676186 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:33.676171    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b7e4826-cc1f-401c-ad81-608f4234cf7f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7e4826-cc1f-401c-ad81-608f4234cf7f-maas-api-tls\") pod \"9b7e4826-cc1f-401c-ad81-608f4234cf7f\" (UID: \"9b7e4826-cc1f-401c-ad81-608f4234cf7f\") "
Oct 30 05:12:33.694032 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:33.679563    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b7e4826-cc1f-401c-ad81-608f4234cf7f-openshift-service-ca" pod "9b7e4826-cc1f-401c-ad81-608f4234cf7f" (UID: "9b7e4826-cc1f-401c-ad81-608f4234cf7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:12:33.713338 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:33.707427    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b7e4826-cc1f-401c-ad81-608f4234cf7f-kube-api-access-kc4v5" pod "9b7e4826-cc1f-401c-ad81-608f4234cf7f" (UID: "9b7e4826-cc1f-401c-ad81-608f4234cf7f"). InnerVolumeSpecName "kube-api-access-kc4v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:12:33.730209 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:33.727381    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b7e4826-cc1f-401c-ad81-608f4234cf7f-maas-api-tls" pod "9b7e4826-cc1f-401c-ad81-608f4234cf7f" (UID: "9b7e4826-cc1f-401c-ad81-608f4234cf7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:12:33.732975 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:33.732930    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b7e4826-cc1f-401c-ad81-608f4234cf7f-metrics-certs" pod "9b7e4826-cc1f-401c-ad81-608f4234cf7f" (UID: "9b7e4826-cc1f-401c-ad81-608f4234cf7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:12:33.783098 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:33.783059    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc4v5\" (UniqueName: \"kubernetes.io/projected/9b7e4826-cc1f-401c-ad81-608f4234cf7f-kube-api-access-kc4v5\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:12:33.783098 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:33.783096    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7e4826-cc1f-401c-ad81-608f4234cf7f-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:12:33.783429 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:33.783109    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b7e4826-cc1f-401c-ad81-608f4234cf7f-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:12:33.783429 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:33.783121    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7e4826-cc1f-401c-ad81-608f4234cf7f-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:12:34.130576 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:34.130545    4874 generic.go:356] "Generic (PLEG): container finished" podID="9b7e4826-cc1f-401c-ad81-608f4234cf7f" containerID="4d607ea1e08474103072523084939bc732dc5767ede2a3f8e9c89f0334b709c6" exitCode=0
Oct 30 05:12:34.130764 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:34.130590    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw" event={"ID":"9b7e4826-cc1f-401c-ad81-608f4234cf7f","Type":"ContainerDied","Data":"4d607ea1e08474103072523084939bc732dc5767ede2a3f8e9c89f0334b709c6"}
Oct 30 05:12:34.130764 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:34.130626    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw" event={"ID":"9b7e4826-cc1f-401c-ad81-608f4234cf7f","Type":"ContainerDied","Data":"dc254fb2780e0851a002e21a86d97e38b908a09b3a08ee4a90c1c98e67f5c2d1"}
Oct 30 05:12:34.130764 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:34.130648    4874 scope.go:122] "RemoveContainer" containerID="4d607ea1e08474103072523084939bc732dc5767ede2a3f8e9c89f0334b709c6"
Oct 30 05:12:34.130764 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:34.130590    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw"
Oct 30 05:12:34.140271 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:34.140227    4874 scope.go:122] "RemoveContainer" containerID="4d607ea1e08474103072523084939bc732dc5767ede2a3f8e9c89f0334b709c6"
Oct 30 05:12:34.140532 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:12:34.140513    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d607ea1e08474103072523084939bc732dc5767ede2a3f8e9c89f0334b709c6\": container with ID starting with 4d607ea1e08474103072523084939bc732dc5767ede2a3f8e9c89f0334b709c6 not found: ID does not exist" containerID="4d607ea1e08474103072523084939bc732dc5767ede2a3f8e9c89f0334b709c6"
Oct 30 05:12:34.140611 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:34.140542    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d607ea1e08474103072523084939bc732dc5767ede2a3f8e9c89f0334b709c6"} err="failed to get container status \"4d607ea1e08474103072523084939bc732dc5767ede2a3f8e9c89f0334b709c6\": rpc error: code = NotFound desc = could not find container \"4d607ea1e08474103072523084939bc732dc5767ede2a3f8e9c89f0334b709c6\": container with ID starting with 4d607ea1e08474103072523084939bc732dc5767ede2a3f8e9c89f0334b709c6 not found: ID does not exist"
Oct 30 05:12:34.179017 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:34.178989    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw"]
Oct 30 05:12:34.182665 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:34.182645    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fcf42a-dbfd85585-cpgbw"]
Oct 30 05:12:34.729455 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:34.729426    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b7e4826-cc1f-401c-ad81-608f4234cf7f" path="/var/lib/kubelet/pods/9b7e4826-cc1f-401c-ad81-608f4234cf7f/volumes"
Oct 30 05:12:34.829765 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:12:34.829736    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:12:55.610147 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:12:55.610108    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:13:01.383774 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:01.383748    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 05:13:01.387147 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:01.387127    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 05:13:05.275972 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:05.275939    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:13:24.440631 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:24.440592    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:13:25.623619 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:13:25.623553    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:13:25.839032 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:25.839000    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"]
Oct 30 05:13:29.677647 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:29.677603    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"]
Oct 30 05:13:31.883502 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:31.883471    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj"]
Oct 30 05:13:31.883932 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:31.883680    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj" podUID="255ff92e-6bed-464a-a995-04cbc315cf99" containerName="maas-api" containerID="cri-o://c6ff7da61618ce5d351e439a45b8c079cecd1bbd0c7f3e9efece40699810d6f4" gracePeriod=30
Oct 30 05:13:32.204812 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:32.204454    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj"
Oct 30 05:13:32.375678 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:32.375640    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/255ff92e-6bed-464a-a995-04cbc315cf99-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/255ff92e-6bed-464a-a995-04cbc315cf99-openshift-service-ca\") pod \"255ff92e-6bed-464a-a995-04cbc315cf99\" (UID: \"255ff92e-6bed-464a-a995-04cbc315cf99\") "
Oct 30 05:13:32.375862 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:32.375843    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/255ff92e-6bed-464a-a995-04cbc315cf99-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/255ff92e-6bed-464a-a995-04cbc315cf99-maas-api-tls\") pod \"255ff92e-6bed-464a-a995-04cbc315cf99\" (UID: \"255ff92e-6bed-464a-a995-04cbc315cf99\") "
Oct 30 05:13:32.375907 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:32.375883    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/255ff92e-6bed-464a-a995-04cbc315cf99-metrics-certs\" (UniqueName: \"kubernetes.io/secret/255ff92e-6bed-464a-a995-04cbc315cf99-metrics-certs\") pod \"255ff92e-6bed-464a-a995-04cbc315cf99\" (UID: \"255ff92e-6bed-464a-a995-04cbc315cf99\") "
Oct 30 05:13:32.375984 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:32.375968    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/255ff92e-6bed-464a-a995-04cbc315cf99-kube-api-access-7575m\" (UniqueName: \"kubernetes.io/projected/255ff92e-6bed-464a-a995-04cbc315cf99-kube-api-access-7575m\") pod \"255ff92e-6bed-464a-a995-04cbc315cf99\" (UID: \"255ff92e-6bed-464a-a995-04cbc315cf99\") "
Oct 30 05:13:32.376107 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:32.376077    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/255ff92e-6bed-464a-a995-04cbc315cf99-openshift-service-ca" pod "255ff92e-6bed-464a-a995-04cbc315cf99" (UID: "255ff92e-6bed-464a-a995-04cbc315cf99"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:13:32.376444 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:32.376416    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/255ff92e-6bed-464a-a995-04cbc315cf99-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:32.378012 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:32.377981    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/255ff92e-6bed-464a-a995-04cbc315cf99-maas-api-tls" pod "255ff92e-6bed-464a-a995-04cbc315cf99" (UID: "255ff92e-6bed-464a-a995-04cbc315cf99"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:13:32.378511 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:32.378488    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/255ff92e-6bed-464a-a995-04cbc315cf99-metrics-certs" pod "255ff92e-6bed-464a-a995-04cbc315cf99" (UID: "255ff92e-6bed-464a-a995-04cbc315cf99"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:13:32.378612 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:32.378500    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/255ff92e-6bed-464a-a995-04cbc315cf99-kube-api-access-7575m" pod "255ff92e-6bed-464a-a995-04cbc315cf99" (UID: "255ff92e-6bed-464a-a995-04cbc315cf99"). InnerVolumeSpecName "kube-api-access-7575m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:13:32.454278 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:32.454218    4874 generic.go:356] "Generic (PLEG): container finished" podID="255ff92e-6bed-464a-a995-04cbc315cf99" containerID="c6ff7da61618ce5d351e439a45b8c079cecd1bbd0c7f3e9efece40699810d6f4" exitCode=0
Oct 30 05:13:32.454428 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:32.454294    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj"
Oct 30 05:13:32.454428 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:32.454304    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj" event={"ID":"255ff92e-6bed-464a-a995-04cbc315cf99","Type":"ContainerDied","Data":"c6ff7da61618ce5d351e439a45b8c079cecd1bbd0c7f3e9efece40699810d6f4"}
Oct 30 05:13:32.454428 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:32.454363    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj" event={"ID":"255ff92e-6bed-464a-a995-04cbc315cf99","Type":"ContainerDied","Data":"9d0915f82e3b810274ae2f8ba8c975c165e9c2500a18acf4bd48459f63cd6935"}
Oct 30 05:13:32.454428 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:32.454387    4874 scope.go:122] "RemoveContainer" containerID="c6ff7da61618ce5d351e439a45b8c079cecd1bbd0c7f3e9efece40699810d6f4"
Oct 30 05:13:32.465584 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:32.465562    4874 scope.go:122] "RemoveContainer" containerID="c6ff7da61618ce5d351e439a45b8c079cecd1bbd0c7f3e9efece40699810d6f4"
Oct 30 05:13:32.465833 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:13:32.465814    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6ff7da61618ce5d351e439a45b8c079cecd1bbd0c7f3e9efece40699810d6f4\": container with ID starting with c6ff7da61618ce5d351e439a45b8c079cecd1bbd0c7f3e9efece40699810d6f4 not found: ID does not exist" containerID="c6ff7da61618ce5d351e439a45b8c079cecd1bbd0c7f3e9efece40699810d6f4"
Oct 30 05:13:32.465886 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:32.465845    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6ff7da61618ce5d351e439a45b8c079cecd1bbd0c7f3e9efece40699810d6f4"} err="failed to get container status \"c6ff7da61618ce5d351e439a45b8c079cecd1bbd0c7f3e9efece40699810d6f4\": rpc error: code = NotFound desc = could not find container \"c6ff7da61618ce5d351e439a45b8c079cecd1bbd0c7f3e9efece40699810d6f4\": container with ID starting with c6ff7da61618ce5d351e439a45b8c079cecd1bbd0c7f3e9efece40699810d6f4 not found: ID does not exist"
Oct 30 05:13:32.477039 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:32.477011    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7575m\" (UniqueName: \"kubernetes.io/projected/255ff92e-6bed-464a-a995-04cbc315cf99-kube-api-access-7575m\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:32.477120 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:32.477042    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/255ff92e-6bed-464a-a995-04cbc315cf99-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:32.477120 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:32.477052    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/255ff92e-6bed-464a-a995-04cbc315cf99-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:32.512479 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:32.512454    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj"]
Oct 30 05:13:32.524092 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:32.524068    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bbce72-67cb6569d6-9vvsj"]
Oct 30 05:13:32.729471 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:32.729441    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="255ff92e-6bed-464a-a995-04cbc315cf99" path="/var/lib/kubelet/pods/255ff92e-6bed-464a-a995-04cbc315cf99/volumes"
Oct 30 05:13:38.661870 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:38.661251    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"]
Oct 30 05:13:38.661870 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:38.661508    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz" podUID="fde6ef20-18b2-4615-952e-8de495f731fe" containerName="istio-proxy" containerID="cri-o://f1640ddf5b217ccb2197286722e084ad3da4af63fc0df8487c3af383acf9d929" gracePeriod=30
Oct 30 05:13:40.859660 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:40.859600    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw" podUID="d605506d-43ec-447c-b573-cb4512867fec" containerName="main" containerID="cri-o://551194fb9431963d868b3fae067fef5fed774de8200128553e8c1af23c94a568" gracePeriod=45
Oct 30 05:13:41.115097 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.115034    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:13:41.241684 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.241647    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-tmp-dir\") pod \"d605506d-43ec-447c-b573-cb4512867fec\" (UID: \"d605506d-43ec-447c-b573-cb4512867fec\") "
Oct 30 05:13:41.241684 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.241685    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d605506d-43ec-447c-b573-cb4512867fec-kube-api-access-zt5ps\" (UniqueName: \"kubernetes.io/projected/d605506d-43ec-447c-b573-cb4512867fec-kube-api-access-zt5ps\") pod \"d605506d-43ec-447c-b573-cb4512867fec\" (UID: \"d605506d-43ec-447c-b573-cb4512867fec\") "
Oct 30 05:13:41.241890 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.241709    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-home\" (UniqueName: \"kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-home\") pod \"d605506d-43ec-447c-b573-cb4512867fec\" (UID: \"d605506d-43ec-447c-b573-cb4512867fec\") "
Oct 30 05:13:41.241890 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.241830    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d605506d-43ec-447c-b573-cb4512867fec-tls-certs\" (UniqueName: \"kubernetes.io/secret/d605506d-43ec-447c-b573-cb4512867fec-tls-certs\") pod \"d605506d-43ec-447c-b573-cb4512867fec\" (UID: \"d605506d-43ec-447c-b573-cb4512867fec\") "
Oct 30 05:13:41.241988 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.241934    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-tmp-dir" pod "d605506d-43ec-447c-b573-cb4512867fec" (UID: "d605506d-43ec-447c-b573-cb4512867fec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:41.241988 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.241951    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-home" pod "d605506d-43ec-447c-b573-cb4512867fec" (UID: "d605506d-43ec-447c-b573-cb4512867fec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:41.241988 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.241946    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-dshm\") pod \"d605506d-43ec-447c-b573-cb4512867fec\" (UID: \"d605506d-43ec-447c-b573-cb4512867fec\") "
Oct 30 05:13:41.242116 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.242008    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-model-cache\") pod \"d605506d-43ec-447c-b573-cb4512867fec\" (UID: \"d605506d-43ec-447c-b573-cb4512867fec\") "
Oct 30 05:13:41.242314 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.242292    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-model-cache" pod "d605506d-43ec-447c-b573-cb4512867fec" (UID: "d605506d-43ec-447c-b573-cb4512867fec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:41.242719 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.242699    4874 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-model-cache\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:41.242769 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.242728    4874 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-tmp-dir\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:41.242769 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.242743    4874 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-home\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:41.243925 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.243898    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-dshm" pod "d605506d-43ec-447c-b573-cb4512867fec" (UID: "d605506d-43ec-447c-b573-cb4512867fec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:41.244062 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.243926    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d605506d-43ec-447c-b573-cb4512867fec-kube-api-access-zt5ps" pod "d605506d-43ec-447c-b573-cb4512867fec" (UID: "d605506d-43ec-447c-b573-cb4512867fec"). InnerVolumeSpecName "kube-api-access-zt5ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:13:41.244062 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.243942    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d605506d-43ec-447c-b573-cb4512867fec-tls-certs" pod "d605506d-43ec-447c-b573-cb4512867fec" (UID: "d605506d-43ec-447c-b573-cb4512867fec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:13:41.344319 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.344280    4874 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d605506d-43ec-447c-b573-cb4512867fec-tls-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:41.344319 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.344310    4874 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d605506d-43ec-447c-b573-cb4512867fec-dshm\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:41.344319 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.344319    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt5ps\" (UniqueName: \"kubernetes.io/projected/d605506d-43ec-447c-b573-cb4512867fec-kube-api-access-zt5ps\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:41.510304 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.510235    4874 generic.go:356] "Generic (PLEG): container finished" podID="d605506d-43ec-447c-b573-cb4512867fec" containerID="551194fb9431963d868b3fae067fef5fed774de8200128553e8c1af23c94a568" exitCode=0
Oct 30 05:13:41.510304 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.510294    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"
Oct 30 05:13:41.510517 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.510334    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw" event={"ID":"d605506d-43ec-447c-b573-cb4512867fec","Type":"ContainerDied","Data":"551194fb9431963d868b3fae067fef5fed774de8200128553e8c1af23c94a568"}
Oct 30 05:13:41.510517 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.510374    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw" event={"ID":"d605506d-43ec-447c-b573-cb4512867fec","Type":"ContainerDied","Data":"3d1165b467f9fae2cfa1565955627de1fac2454ec75a41ecaa7a1c9d45b035e5"}
Oct 30 05:13:41.510517 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.510390    4874 scope.go:122] "RemoveContainer" containerID="551194fb9431963d868b3fae067fef5fed774de8200128553e8c1af23c94a568"
Oct 30 05:13:41.519850 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.519833    4874 scope.go:122] "RemoveContainer" containerID="551194fb9431963d868b3fae067fef5fed774de8200128553e8c1af23c94a568"
Oct 30 05:13:41.520098 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:13:41.520082    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"551194fb9431963d868b3fae067fef5fed774de8200128553e8c1af23c94a568\": container with ID starting with 551194fb9431963d868b3fae067fef5fed774de8200128553e8c1af23c94a568 not found: ID does not exist" containerID="551194fb9431963d868b3fae067fef5fed774de8200128553e8c1af23c94a568"
Oct 30 05:13:41.520150 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.520105    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"551194fb9431963d868b3fae067fef5fed774de8200128553e8c1af23c94a568"} err="failed to get container status \"551194fb9431963d868b3fae067fef5fed774de8200128553e8c1af23c94a568\": rpc error: code = NotFound desc = could not find container \"551194fb9431963d868b3fae067fef5fed774de8200128553e8c1af23c94a568\": container with ID starting with 551194fb9431963d868b3fae067fef5fed774de8200128553e8c1af23c94a568 not found: ID does not exist"
Oct 30 05:13:41.594247 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.594218    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"]
Oct 30 05:13:41.605059 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:41.605037    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-e6b837/sub-test-model-e6b837-kserve-8694db69c9-w52jw"]
Oct 30 05:13:42.729695 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:42.729665    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d605506d-43ec-447c-b573-cb4512867fec" path="/var/lib/kubelet/pods/d605506d-43ec-447c-b573-cb4512867fec/volumes"
Oct 30 05:13:43.923486 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:43.923464    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 05:13:44.089572 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.089544    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-workload-socket\") pod \"fde6ef20-18b2-4615-952e-8de495f731fe\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") "
Oct 30 05:13:44.089770 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.089595    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fde6ef20-18b2-4615-952e-8de495f731fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fde6ef20-18b2-4615-952e-8de495f731fe-istiod-ca-cert\") pod \"fde6ef20-18b2-4615-952e-8de495f731fe\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") "
Oct 30 05:13:44.089770 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.089658    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-istio-data\") pod \"fde6ef20-18b2-4615-952e-8de495f731fe\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") "
Oct 30 05:13:44.089770 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.089705    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-istio-envoy\") pod \"fde6ef20-18b2-4615-952e-8de495f731fe\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") "
Oct 30 05:13:44.089770 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.089768    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-credential-socket\") pod \"fde6ef20-18b2-4615-952e-8de495f731fe\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") "
Oct 30 05:13:44.090003 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.089797    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fde6ef20-18b2-4615-952e-8de495f731fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fde6ef20-18b2-4615-952e-8de495f731fe-istio-podinfo\") pod \"fde6ef20-18b2-4615-952e-8de495f731fe\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") "
Oct 30 05:13:44.090003 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.089823    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fde6ef20-18b2-4615-952e-8de495f731fe-istio-token\" (UniqueName: \"kubernetes.io/projected/fde6ef20-18b2-4615-952e-8de495f731fe-istio-token\") pod \"fde6ef20-18b2-4615-952e-8de495f731fe\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") "
Oct 30 05:13:44.090003 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.089861    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-workload-socket" pod "fde6ef20-18b2-4615-952e-8de495f731fe" (UID: "fde6ef20-18b2-4615-952e-8de495f731fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:44.090003 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.089877    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-workload-certs\") pod \"fde6ef20-18b2-4615-952e-8de495f731fe\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") "
Oct 30 05:13:44.090003 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.089970    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fde6ef20-18b2-4615-952e-8de495f731fe-kube-api-access-rx5kh\" (UniqueName: \"kubernetes.io/projected/fde6ef20-18b2-4615-952e-8de495f731fe-kube-api-access-rx5kh\") pod \"fde6ef20-18b2-4615-952e-8de495f731fe\" (UID: \"fde6ef20-18b2-4615-952e-8de495f731fe\") "
Oct 30 05:13:44.090003 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.089999    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-istio-data" pod "fde6ef20-18b2-4615-952e-8de495f731fe" (UID: "fde6ef20-18b2-4615-952e-8de495f731fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:44.090340 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.090090    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-credential-socket" pod "fde6ef20-18b2-4615-952e-8de495f731fe" (UID: "fde6ef20-18b2-4615-952e-8de495f731fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:44.090340 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.090135    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fde6ef20-18b2-4615-952e-8de495f731fe-istiod-ca-cert" pod "fde6ef20-18b2-4615-952e-8de495f731fe" (UID: "fde6ef20-18b2-4615-952e-8de495f731fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:13:44.090340 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.090154    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-workload-certs" pod "fde6ef20-18b2-4615-952e-8de495f731fe" (UID: "fde6ef20-18b2-4615-952e-8de495f731fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:44.090503 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.090486    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fde6ef20-18b2-4615-952e-8de495f731fe-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:44.090551 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.090506    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:44.090551 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.090519    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:44.090551 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.090532    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:44.090551 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.090544    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:44.091764 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.091737    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-istio-envoy" pod "fde6ef20-18b2-4615-952e-8de495f731fe" (UID: "fde6ef20-18b2-4615-952e-8de495f731fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:44.091880 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.091834    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fde6ef20-18b2-4615-952e-8de495f731fe-istio-podinfo" pod "fde6ef20-18b2-4615-952e-8de495f731fe" (UID: "fde6ef20-18b2-4615-952e-8de495f731fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:13:44.092344 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.092325    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde6ef20-18b2-4615-952e-8de495f731fe-kube-api-access-rx5kh" pod "fde6ef20-18b2-4615-952e-8de495f731fe" (UID: "fde6ef20-18b2-4615-952e-8de495f731fe"). InnerVolumeSpecName "kube-api-access-rx5kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:13:44.092418 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.092365    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde6ef20-18b2-4615-952e-8de495f731fe-istio-token" pod "fde6ef20-18b2-4615-952e-8de495f731fe" (UID: "fde6ef20-18b2-4615-952e-8de495f731fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:13:44.125015 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.124987    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"]
Oct 30 05:13:44.191562 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.191534    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fde6ef20-18b2-4615-952e-8de495f731fe-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:44.191562 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.191559    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fde6ef20-18b2-4615-952e-8de495f731fe-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:44.191562 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.191567    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fde6ef20-18b2-4615-952e-8de495f731fe-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:44.191769 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.191577    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx5kh\" (UniqueName: \"kubernetes.io/projected/fde6ef20-18b2-4615-952e-8de495f731fe-kube-api-access-rx5kh\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:44.528268 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.528222    4874 generic.go:356] "Generic (PLEG): container finished" podID="fde6ef20-18b2-4615-952e-8de495f731fe" containerID="f1640ddf5b217ccb2197286722e084ad3da4af63fc0df8487c3af383acf9d929" exitCode=0
Oct 30 05:13:44.528469 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.528287    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"
Oct 30 05:13:44.528469 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.528279    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz" event={"ID":"fde6ef20-18b2-4615-952e-8de495f731fe","Type":"ContainerDied","Data":"f1640ddf5b217ccb2197286722e084ad3da4af63fc0df8487c3af383acf9d929"}
Oct 30 05:13:44.528469 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.528413    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz" event={"ID":"fde6ef20-18b2-4615-952e-8de495f731fe","Type":"ContainerDied","Data":"dc9d3cd0bc646b6afad93648b72085a85c95b5c112f232dc1c344ab272da6d51"}
Oct 30 05:13:44.528469 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.528443    4874 scope.go:122] "RemoveContainer" containerID="f1640ddf5b217ccb2197286722e084ad3da4af63fc0df8487c3af383acf9d929"
Oct 30 05:13:44.538539 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.538522    4874 scope.go:122] "RemoveContainer" containerID="bbecf76d40321ecb65e5879d73226a56396305b68f8ac400315abba2fa5b4c30"
Oct 30 05:13:44.546873 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.546853    4874 scope.go:122] "RemoveContainer" containerID="f1640ddf5b217ccb2197286722e084ad3da4af63fc0df8487c3af383acf9d929"
Oct 30 05:13:44.547122 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:13:44.547102    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1640ddf5b217ccb2197286722e084ad3da4af63fc0df8487c3af383acf9d929\": container with ID starting with f1640ddf5b217ccb2197286722e084ad3da4af63fc0df8487c3af383acf9d929 not found: ID does not exist" containerID="f1640ddf5b217ccb2197286722e084ad3da4af63fc0df8487c3af383acf9d929"
Oct 30 05:13:44.547197 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.547132    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1640ddf5b217ccb2197286722e084ad3da4af63fc0df8487c3af383acf9d929"} err="failed to get container status \"f1640ddf5b217ccb2197286722e084ad3da4af63fc0df8487c3af383acf9d929\": rpc error: code = NotFound desc = could not find container \"f1640ddf5b217ccb2197286722e084ad3da4af63fc0df8487c3af383acf9d929\": container with ID starting with f1640ddf5b217ccb2197286722e084ad3da4af63fc0df8487c3af383acf9d929 not found: ID does not exist"
Oct 30 05:13:44.547197 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.547154    4874 scope.go:122] "RemoveContainer" containerID="bbecf76d40321ecb65e5879d73226a56396305b68f8ac400315abba2fa5b4c30"
Oct 30 05:13:44.547420 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:13:44.547402    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbecf76d40321ecb65e5879d73226a56396305b68f8ac400315abba2fa5b4c30\": container with ID starting with bbecf76d40321ecb65e5879d73226a56396305b68f8ac400315abba2fa5b4c30 not found: ID does not exist" containerID="bbecf76d40321ecb65e5879d73226a56396305b68f8ac400315abba2fa5b4c30"
Oct 30 05:13:44.547459 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.547428    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbecf76d40321ecb65e5879d73226a56396305b68f8ac400315abba2fa5b4c30"} err="failed to get container status \"bbecf76d40321ecb65e5879d73226a56396305b68f8ac400315abba2fa5b4c30\": rpc error: code = NotFound desc = could not find container \"bbecf76d40321ecb65e5879d73226a56396305b68f8ac400315abba2fa5b4c30\": container with ID starting with bbecf76d40321ecb65e5879d73226a56396305b68f8ac400315abba2fa5b4c30 not found: ID does not exist"
Oct 30 05:13:44.578568 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.578546    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"]
Oct 30 05:13:44.590035 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.590017    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-bbce72-openshift-default-6bb7f5c447-dzvnz"]
Oct 30 05:13:44.701283 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.701250    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x" podUID="d363d951-949c-4438-8c54-e65908025cfd" containerName="main" containerID="cri-o://fde13ad63db0aa367fdd79f947bbf8a601b2f15e65b891773463fa1acc0fbbbc" gracePeriod=45
Oct 30 05:13:44.729336 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.729313    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fde6ef20-18b2-4615-952e-8de495f731fe" path="/var/lib/kubelet/pods/fde6ef20-18b2-4615-952e-8de495f731fe/volumes"
Oct 30 05:13:44.962636 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:44.962611    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:13:45.103043 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.102965    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-tmp-dir\") pod \"d363d951-949c-4438-8c54-e65908025cfd\" (UID: \"d363d951-949c-4438-8c54-e65908025cfd\") "
Oct 30 05:13:45.103189 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.103098    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-dshm\") pod \"d363d951-949c-4438-8c54-e65908025cfd\" (UID: \"d363d951-949c-4438-8c54-e65908025cfd\") "
Oct 30 05:13:45.103189 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.103129    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d363d951-949c-4438-8c54-e65908025cfd-tls-certs\" (UniqueName: \"kubernetes.io/secret/d363d951-949c-4438-8c54-e65908025cfd-tls-certs\") pod \"d363d951-949c-4438-8c54-e65908025cfd\" (UID: \"d363d951-949c-4438-8c54-e65908025cfd\") "
Oct 30 05:13:45.103189 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.103162    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d363d951-949c-4438-8c54-e65908025cfd-kube-api-access-lc22b\" (UniqueName: \"kubernetes.io/projected/d363d951-949c-4438-8c54-e65908025cfd-kube-api-access-lc22b\") pod \"d363d951-949c-4438-8c54-e65908025cfd\" (UID: \"d363d951-949c-4438-8c54-e65908025cfd\") "
Oct 30 05:13:45.103402 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.103208    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-home\" (UniqueName: \"kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-home\") pod \"d363d951-949c-4438-8c54-e65908025cfd\" (UID: \"d363d951-949c-4438-8c54-e65908025cfd\") "
Oct 30 05:13:45.103402 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.103219    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-tmp-dir" pod "d363d951-949c-4438-8c54-e65908025cfd" (UID: "d363d951-949c-4438-8c54-e65908025cfd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:45.103516 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.103409    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-model-cache\") pod \"d363d951-949c-4438-8c54-e65908025cfd\" (UID: \"d363d951-949c-4438-8c54-e65908025cfd\") "
Oct 30 05:13:45.103516 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.103468    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-home" pod "d363d951-949c-4438-8c54-e65908025cfd" (UID: "d363d951-949c-4438-8c54-e65908025cfd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:45.103761 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.103740    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-model-cache" pod "d363d951-949c-4438-8c54-e65908025cfd" (UID: "d363d951-949c-4438-8c54-e65908025cfd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:45.103857 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.103838    4874 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-home\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:45.103911 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.103857    4874 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-model-cache\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:45.103911 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.103875    4874 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-tmp-dir\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:45.105562 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.105537    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d363d951-949c-4438-8c54-e65908025cfd-kube-api-access-lc22b" pod "d363d951-949c-4438-8c54-e65908025cfd" (UID: "d363d951-949c-4438-8c54-e65908025cfd"). InnerVolumeSpecName "kube-api-access-lc22b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:13:45.105678 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.105662    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d363d951-949c-4438-8c54-e65908025cfd-tls-certs" pod "d363d951-949c-4438-8c54-e65908025cfd" (UID: "d363d951-949c-4438-8c54-e65908025cfd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:13:45.105760 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.105743    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-dshm" pod "d363d951-949c-4438-8c54-e65908025cfd" (UID: "d363d951-949c-4438-8c54-e65908025cfd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:45.204577 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.204547    4874 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d363d951-949c-4438-8c54-e65908025cfd-dshm\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:45.204577 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.204573    4874 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d363d951-949c-4438-8c54-e65908025cfd-tls-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:45.204577 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.204582    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc22b\" (UniqueName: \"kubernetes.io/projected/d363d951-949c-4438-8c54-e65908025cfd-kube-api-access-lc22b\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:45.534938 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.534902    4874 generic.go:356] "Generic (PLEG): container finished" podID="d363d951-949c-4438-8c54-e65908025cfd" containerID="fde13ad63db0aa367fdd79f947bbf8a601b2f15e65b891773463fa1acc0fbbbc" exitCode=0
Oct 30 05:13:45.535111 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.534950    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"
Oct 30 05:13:45.535111 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.534983    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x" event={"ID":"d363d951-949c-4438-8c54-e65908025cfd","Type":"ContainerDied","Data":"fde13ad63db0aa367fdd79f947bbf8a601b2f15e65b891773463fa1acc0fbbbc"}
Oct 30 05:13:45.535111 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.535026    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x" event={"ID":"d363d951-949c-4438-8c54-e65908025cfd","Type":"ContainerDied","Data":"d6114bb3f10a8c3826ce42dba1eeb9b177753d7445be38701f058a7199bffaea"}
Oct 30 05:13:45.535111 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.535046    4874 scope.go:122] "RemoveContainer" containerID="fde13ad63db0aa367fdd79f947bbf8a601b2f15e65b891773463fa1acc0fbbbc"
Oct 30 05:13:45.544080 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.544063    4874 scope.go:122] "RemoveContainer" containerID="fde13ad63db0aa367fdd79f947bbf8a601b2f15e65b891773463fa1acc0fbbbc"
Oct 30 05:13:45.544383 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:13:45.544363    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fde13ad63db0aa367fdd79f947bbf8a601b2f15e65b891773463fa1acc0fbbbc\": container with ID starting with fde13ad63db0aa367fdd79f947bbf8a601b2f15e65b891773463fa1acc0fbbbc not found: ID does not exist" containerID="fde13ad63db0aa367fdd79f947bbf8a601b2f15e65b891773463fa1acc0fbbbc"
Oct 30 05:13:45.544450 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.544391    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fde13ad63db0aa367fdd79f947bbf8a601b2f15e65b891773463fa1acc0fbbbc"} err="failed to get container status \"fde13ad63db0aa367fdd79f947bbf8a601b2f15e65b891773463fa1acc0fbbbc\": rpc error: code = NotFound desc = could not find container \"fde13ad63db0aa367fdd79f947bbf8a601b2f15e65b891773463fa1acc0fbbbc\": container with ID starting with fde13ad63db0aa367fdd79f947bbf8a601b2f15e65b891773463fa1acc0fbbbc not found: ID does not exist"
Oct 30 05:13:45.584029 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.583999    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"]
Oct 30 05:13:45.595630 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:45.595595    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-247158/sub-test-model-247158-kserve-c47cff-qqs9x"]
Oct 30 05:13:46.729308 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:46.729274    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d363d951-949c-4438-8c54-e65908025cfd" path="/var/lib/kubelet/pods/d363d951-949c-4438-8c54-e65908025cfd/volumes"
Oct 30 05:13:48.016849 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:48.016820    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4"]
Oct 30 05:13:48.017620 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:48.017589    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4" podUID="142fff76-0542-40d8-bea6-f46a50366b1d" containerName="maas-api" containerID="cri-o://5b341ddf58e43603995f97fbfc2996b6f629a00c846c5ae0266698043f6fdf46" gracePeriod=30
Oct 30 05:13:48.339162 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:48.339129    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4"
Oct 30 05:13:48.445270 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:48.443662    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/142fff76-0542-40d8-bea6-f46a50366b1d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/142fff76-0542-40d8-bea6-f46a50366b1d-metrics-certs\") pod \"142fff76-0542-40d8-bea6-f46a50366b1d\" (UID: \"142fff76-0542-40d8-bea6-f46a50366b1d\") "
Oct 30 05:13:48.445270 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:48.443823    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/142fff76-0542-40d8-bea6-f46a50366b1d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/142fff76-0542-40d8-bea6-f46a50366b1d-openshift-service-ca\") pod \"142fff76-0542-40d8-bea6-f46a50366b1d\" (UID: \"142fff76-0542-40d8-bea6-f46a50366b1d\") "
Oct 30 05:13:48.445270 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:48.444002    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/142fff76-0542-40d8-bea6-f46a50366b1d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/142fff76-0542-40d8-bea6-f46a50366b1d-maas-api-tls\") pod \"142fff76-0542-40d8-bea6-f46a50366b1d\" (UID: \"142fff76-0542-40d8-bea6-f46a50366b1d\") "
Oct 30 05:13:48.445270 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:48.444199    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/142fff76-0542-40d8-bea6-f46a50366b1d-kube-api-access-6vdj7\" (UniqueName: \"kubernetes.io/projected/142fff76-0542-40d8-bea6-f46a50366b1d-kube-api-access-6vdj7\") pod \"142fff76-0542-40d8-bea6-f46a50366b1d\" (UID: \"142fff76-0542-40d8-bea6-f46a50366b1d\") "
Oct 30 05:13:48.470431 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:48.461424    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/142fff76-0542-40d8-bea6-f46a50366b1d-openshift-service-ca" pod "142fff76-0542-40d8-bea6-f46a50366b1d" (UID: "142fff76-0542-40d8-bea6-f46a50366b1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:13:48.470431 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:48.466416    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/142fff76-0542-40d8-bea6-f46a50366b1d-kube-api-access-6vdj7" pod "142fff76-0542-40d8-bea6-f46a50366b1d" (UID: "142fff76-0542-40d8-bea6-f46a50366b1d"). InnerVolumeSpecName "kube-api-access-6vdj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:13:48.475791 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:48.475326    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/142fff76-0542-40d8-bea6-f46a50366b1d-metrics-certs" pod "142fff76-0542-40d8-bea6-f46a50366b1d" (UID: "142fff76-0542-40d8-bea6-f46a50366b1d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:13:48.478857 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:48.478835    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/142fff76-0542-40d8-bea6-f46a50366b1d-maas-api-tls" pod "142fff76-0542-40d8-bea6-f46a50366b1d" (UID: "142fff76-0542-40d8-bea6-f46a50366b1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:13:48.545564 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:48.545491    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/142fff76-0542-40d8-bea6-f46a50366b1d-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:48.545564 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:48.545517    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/142fff76-0542-40d8-bea6-f46a50366b1d-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:48.545564 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:48.545527    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vdj7\" (UniqueName: \"kubernetes.io/projected/142fff76-0542-40d8-bea6-f46a50366b1d-kube-api-access-6vdj7\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:48.545564 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:48.545536    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/142fff76-0542-40d8-bea6-f46a50366b1d-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:48.554925 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:48.554893    4874 generic.go:356] "Generic (PLEG): container finished" podID="142fff76-0542-40d8-bea6-f46a50366b1d" containerID="5b341ddf58e43603995f97fbfc2996b6f629a00c846c5ae0266698043f6fdf46" exitCode=0
Oct 30 05:13:48.555060 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:48.554943    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4"
Oct 30 05:13:48.555060 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:48.554976    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4" event={"ID":"142fff76-0542-40d8-bea6-f46a50366b1d","Type":"ContainerDied","Data":"5b341ddf58e43603995f97fbfc2996b6f629a00c846c5ae0266698043f6fdf46"}
Oct 30 05:13:48.555060 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:48.555043    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4" event={"ID":"142fff76-0542-40d8-bea6-f46a50366b1d","Type":"ContainerDied","Data":"ec09339352a110e83b50387f7d9aa72b9f6afbb194996f1c927ee45da6deadf5"}
Oct 30 05:13:48.555190 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:48.555071    4874 scope.go:122] "RemoveContainer" containerID="5b341ddf58e43603995f97fbfc2996b6f629a00c846c5ae0266698043f6fdf46"
Oct 30 05:13:48.565425 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:48.565407    4874 scope.go:122] "RemoveContainer" containerID="5b341ddf58e43603995f97fbfc2996b6f629a00c846c5ae0266698043f6fdf46"
Oct 30 05:13:48.565649 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:13:48.565631    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b341ddf58e43603995f97fbfc2996b6f629a00c846c5ae0266698043f6fdf46\": container with ID starting with 5b341ddf58e43603995f97fbfc2996b6f629a00c846c5ae0266698043f6fdf46 not found: ID does not exist" containerID="5b341ddf58e43603995f97fbfc2996b6f629a00c846c5ae0266698043f6fdf46"
Oct 30 05:13:48.565702 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:48.565657    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b341ddf58e43603995f97fbfc2996b6f629a00c846c5ae0266698043f6fdf46"} err="failed to get container status \"5b341ddf58e43603995f97fbfc2996b6f629a00c846c5ae0266698043f6fdf46\": rpc error: code = NotFound desc = could not find container \"5b341ddf58e43603995f97fbfc2996b6f629a00c846c5ae0266698043f6fdf46\": container with ID starting with 5b341ddf58e43603995f97fbfc2996b6f629a00c846c5ae0266698043f6fdf46 not found: ID does not exist"
Oct 30 05:13:48.604087 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:48.604063    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4"]
Oct 30 05:13:48.606227 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:48.606206    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c0e964-66db5bd589-r9bn4"]
Oct 30 05:13:48.728111 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:48.728081    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="142fff76-0542-40d8-bea6-f46a50366b1d" path="/var/lib/kubelet/pods/142fff76-0542-40d8-bea6-f46a50366b1d/volumes"
Oct 30 05:13:55.609645 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:13:55.609604    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:13:56.214542 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:56.214507    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:13:59.144650 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.144611    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8" podUID="47a05334-0a58-4dd9-a9e3-2194031456cd" containerName="main" containerID="cri-o://88ea629527c40f3ba8f6bf9aa6c2687a41b1dd82e253e759f61c3011aec823cf" gracePeriod=45
Oct 30 05:13:59.429709 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.429684    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 05:13:59.482807 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.482766    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47a05334-0a58-4dd9-a9e3-2194031456cd-tls-certs\" (UniqueName: \"kubernetes.io/secret/47a05334-0a58-4dd9-a9e3-2194031456cd-tls-certs\") pod \"47a05334-0a58-4dd9-a9e3-2194031456cd\" (UID: \"47a05334-0a58-4dd9-a9e3-2194031456cd\") "
Oct 30 05:13:59.483431 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.483409    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-dshm\") pod \"47a05334-0a58-4dd9-a9e3-2194031456cd\" (UID: \"47a05334-0a58-4dd9-a9e3-2194031456cd\") "
Oct 30 05:13:59.483879 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.483575    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-tmp-dir\") pod \"47a05334-0a58-4dd9-a9e3-2194031456cd\" (UID: \"47a05334-0a58-4dd9-a9e3-2194031456cd\") "
Oct 30 05:13:59.483879 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.483844    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-tmp-dir" pod "47a05334-0a58-4dd9-a9e3-2194031456cd" (UID: "47a05334-0a58-4dd9-a9e3-2194031456cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:59.484381 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.484095    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-home\" (UniqueName: \"kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-home\") pod \"47a05334-0a58-4dd9-a9e3-2194031456cd\" (UID: \"47a05334-0a58-4dd9-a9e3-2194031456cd\") "
Oct 30 05:13:59.484381 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.484343    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-home" pod "47a05334-0a58-4dd9-a9e3-2194031456cd" (UID: "47a05334-0a58-4dd9-a9e3-2194031456cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:59.484761 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.484737    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-model-cache\") pod \"47a05334-0a58-4dd9-a9e3-2194031456cd\" (UID: \"47a05334-0a58-4dd9-a9e3-2194031456cd\") "
Oct 30 05:13:59.484883 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.484824    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47a05334-0a58-4dd9-a9e3-2194031456cd-kube-api-access-qpxvr\" (UniqueName: \"kubernetes.io/projected/47a05334-0a58-4dd9-a9e3-2194031456cd-kube-api-access-qpxvr\") pod \"47a05334-0a58-4dd9-a9e3-2194031456cd\" (UID: \"47a05334-0a58-4dd9-a9e3-2194031456cd\") "
Oct 30 05:13:59.485363 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.485185    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-model-cache" pod "47a05334-0a58-4dd9-a9e3-2194031456cd" (UID: "47a05334-0a58-4dd9-a9e3-2194031456cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:59.485363 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.485305    4874 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-tmp-dir\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:59.485363 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.485324    4874 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-home\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:59.485363 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.485337    4874 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-model-cache\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:59.486392 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.486368    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47a05334-0a58-4dd9-a9e3-2194031456cd-tls-certs" pod "47a05334-0a58-4dd9-a9e3-2194031456cd" (UID: "47a05334-0a58-4dd9-a9e3-2194031456cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:13:59.486499 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.486337    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-dshm" pod "47a05334-0a58-4dd9-a9e3-2194031456cd" (UID: "47a05334-0a58-4dd9-a9e3-2194031456cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:13:59.489584 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.489557    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a05334-0a58-4dd9-a9e3-2194031456cd-kube-api-access-qpxvr" pod "47a05334-0a58-4dd9-a9e3-2194031456cd" (UID: "47a05334-0a58-4dd9-a9e3-2194031456cd"). InnerVolumeSpecName "kube-api-access-qpxvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:13:59.585960 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.585922    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpxvr\" (UniqueName: \"kubernetes.io/projected/47a05334-0a58-4dd9-a9e3-2194031456cd-kube-api-access-qpxvr\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:59.585960 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.585954    4874 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47a05334-0a58-4dd9-a9e3-2194031456cd-tls-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:59.585960 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.585964    4874 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47a05334-0a58-4dd9-a9e3-2194031456cd-dshm\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:13:59.640648 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.640616    4874 generic.go:356] "Generic (PLEG): container finished" podID="47a05334-0a58-4dd9-a9e3-2194031456cd" containerID="88ea629527c40f3ba8f6bf9aa6c2687a41b1dd82e253e759f61c3011aec823cf" exitCode=0
Oct 30 05:13:59.640828 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.640663    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"
Oct 30 05:13:59.640828 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.640708    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8" event={"ID":"47a05334-0a58-4dd9-a9e3-2194031456cd","Type":"ContainerDied","Data":"88ea629527c40f3ba8f6bf9aa6c2687a41b1dd82e253e759f61c3011aec823cf"}
Oct 30 05:13:59.640828 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.640762    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8" event={"ID":"47a05334-0a58-4dd9-a9e3-2194031456cd","Type":"ContainerDied","Data":"7cae1443273757d55a1ffceba833d72189204819b2c41bff8b02717a3f790e65"}
Oct 30 05:13:59.640828 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.640784    4874 scope.go:122] "RemoveContainer" containerID="88ea629527c40f3ba8f6bf9aa6c2687a41b1dd82e253e759f61c3011aec823cf"
Oct 30 05:13:59.650901 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.650882    4874 scope.go:122] "RemoveContainer" containerID="88ea629527c40f3ba8f6bf9aa6c2687a41b1dd82e253e759f61c3011aec823cf"
Oct 30 05:13:59.651178 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:13:59.651158    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ea629527c40f3ba8f6bf9aa6c2687a41b1dd82e253e759f61c3011aec823cf\": container with ID starting with 88ea629527c40f3ba8f6bf9aa6c2687a41b1dd82e253e759f61c3011aec823cf not found: ID does not exist" containerID="88ea629527c40f3ba8f6bf9aa6c2687a41b1dd82e253e759f61c3011aec823cf"
Oct 30 05:13:59.651229 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.651186    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ea629527c40f3ba8f6bf9aa6c2687a41b1dd82e253e759f61c3011aec823cf"} err="failed to get container status \"88ea629527c40f3ba8f6bf9aa6c2687a41b1dd82e253e759f61c3011aec823cf\": rpc error: code = NotFound desc = could not find container \"88ea629527c40f3ba8f6bf9aa6c2687a41b1dd82e253e759f61c3011aec823cf\": container with ID starting with 88ea629527c40f3ba8f6bf9aa6c2687a41b1dd82e253e759f61c3011aec823cf not found: ID does not exist"
Oct 30 05:13:59.690217 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.690154    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"]
Oct 30 05:13:59.693918 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:13:59.693891    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-a2f984/distinct-w4-a2f984-kserve-54b7989d4c-74bg8"]
Oct 30 05:14:00.728948 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:00.728916    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47a05334-0a58-4dd9-a9e3-2194031456cd" path="/var/lib/kubelet/pods/47a05334-0a58-4dd9-a9e3-2194031456cd/volumes"
Oct 30 05:14:08.537676 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:08.537620    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:14:10.809112 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:10.809072    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"]
Oct 30 05:14:10.818696 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:10.818655    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw" podUID="c4ec0ec0-354e-41f9-897e-ecbd08404b2e" containerName="istio-proxy" containerID="cri-o://00736f477cb56865fbc045a7404d67a6c67ac5bf01df7e910b151a1b8850e8d2" gracePeriod=30
Oct 30 05:14:12.711692 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:12.711658    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh"]
Oct 30 05:14:12.712228 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:12.711875    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh" podUID="7479a6e1-f414-411e-b1a0-a56fe33f8262" containerName="maas-api" containerID="cri-o://a2bbc54cb3fbf6aed2870e3a6214f89e95f7a25618d0f2a1c8e36199c1f8fa59" gracePeriod=30
Oct 30 05:14:12.961332 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:12.959923    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh"
Oct 30 05:14:13.015547 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:13.006198    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7479a6e1-f414-411e-b1a0-a56fe33f8262-kube-api-access-6nh5b\" (UniqueName: \"kubernetes.io/projected/7479a6e1-f414-411e-b1a0-a56fe33f8262-kube-api-access-6nh5b\") pod \"7479a6e1-f414-411e-b1a0-a56fe33f8262\" (UID: \"7479a6e1-f414-411e-b1a0-a56fe33f8262\") "
Oct 30 05:14:13.015547 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:13.006275    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7479a6e1-f414-411e-b1a0-a56fe33f8262-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7479a6e1-f414-411e-b1a0-a56fe33f8262-openshift-service-ca\") pod \"7479a6e1-f414-411e-b1a0-a56fe33f8262\" (UID: \"7479a6e1-f414-411e-b1a0-a56fe33f8262\") "
Oct 30 05:14:13.015547 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:13.006426    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7479a6e1-f414-411e-b1a0-a56fe33f8262-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7479a6e1-f414-411e-b1a0-a56fe33f8262-metrics-certs\") pod \"7479a6e1-f414-411e-b1a0-a56fe33f8262\" (UID: \"7479a6e1-f414-411e-b1a0-a56fe33f8262\") "
Oct 30 05:14:13.015547 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:13.006483    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7479a6e1-f414-411e-b1a0-a56fe33f8262-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7479a6e1-f414-411e-b1a0-a56fe33f8262-maas-api-tls\") pod \"7479a6e1-f414-411e-b1a0-a56fe33f8262\" (UID: \"7479a6e1-f414-411e-b1a0-a56fe33f8262\") "
Oct 30 05:14:13.015547 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:13.014151    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7479a6e1-f414-411e-b1a0-a56fe33f8262-openshift-service-ca" pod "7479a6e1-f414-411e-b1a0-a56fe33f8262" (UID: "7479a6e1-f414-411e-b1a0-a56fe33f8262"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:14:13.019015 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:13.015875    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7479a6e1-f414-411e-b1a0-a56fe33f8262-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:13.050113 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:13.037345    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7479a6e1-f414-411e-b1a0-a56fe33f8262-metrics-certs" pod "7479a6e1-f414-411e-b1a0-a56fe33f8262" (UID: "7479a6e1-f414-411e-b1a0-a56fe33f8262"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:14:13.055865 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:13.055825    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7479a6e1-f414-411e-b1a0-a56fe33f8262-kube-api-access-6nh5b" pod "7479a6e1-f414-411e-b1a0-a56fe33f8262" (UID: "7479a6e1-f414-411e-b1a0-a56fe33f8262"). InnerVolumeSpecName "kube-api-access-6nh5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:13.058396 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:13.058363    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7479a6e1-f414-411e-b1a0-a56fe33f8262-maas-api-tls" pod "7479a6e1-f414-411e-b1a0-a56fe33f8262" (UID: "7479a6e1-f414-411e-b1a0-a56fe33f8262"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:14:13.116655 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:13.116628    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nh5b\" (UniqueName: \"kubernetes.io/projected/7479a6e1-f414-411e-b1a0-a56fe33f8262-kube-api-access-6nh5b\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:13.116655 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:13.116652    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7479a6e1-f414-411e-b1a0-a56fe33f8262-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:13.116794 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:13.116661    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7479a6e1-f414-411e-b1a0-a56fe33f8262-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:13.736413 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:13.736383    4874 generic.go:356] "Generic (PLEG): container finished" podID="7479a6e1-f414-411e-b1a0-a56fe33f8262" containerID="a2bbc54cb3fbf6aed2870e3a6214f89e95f7a25618d0f2a1c8e36199c1f8fa59" exitCode=0
Oct 30 05:14:13.736899 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:13.736427    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh"
Oct 30 05:14:13.736899 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:13.736465    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh" event={"ID":"7479a6e1-f414-411e-b1a0-a56fe33f8262","Type":"ContainerDied","Data":"a2bbc54cb3fbf6aed2870e3a6214f89e95f7a25618d0f2a1c8e36199c1f8fa59"}
Oct 30 05:14:13.736899 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:13.736526    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh" event={"ID":"7479a6e1-f414-411e-b1a0-a56fe33f8262","Type":"ContainerDied","Data":"39315431dbc66f22df94027dfdfa962675cb5558bfa7fbb37972d96c5ac34773"}
Oct 30 05:14:13.736899 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:13.736556    4874 scope.go:122] "RemoveContainer" containerID="a2bbc54cb3fbf6aed2870e3a6214f89e95f7a25618d0f2a1c8e36199c1f8fa59"
Oct 30 05:14:13.747080 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:13.747063    4874 scope.go:122] "RemoveContainer" containerID="a2bbc54cb3fbf6aed2870e3a6214f89e95f7a25618d0f2a1c8e36199c1f8fa59"
Oct 30 05:14:13.747360 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:14:13.747341    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2bbc54cb3fbf6aed2870e3a6214f89e95f7a25618d0f2a1c8e36199c1f8fa59\": container with ID starting with a2bbc54cb3fbf6aed2870e3a6214f89e95f7a25618d0f2a1c8e36199c1f8fa59 not found: ID does not exist" containerID="a2bbc54cb3fbf6aed2870e3a6214f89e95f7a25618d0f2a1c8e36199c1f8fa59"
Oct 30 05:14:13.747425 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:13.747370    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2bbc54cb3fbf6aed2870e3a6214f89e95f7a25618d0f2a1c8e36199c1f8fa59"} err="failed to get container status \"a2bbc54cb3fbf6aed2870e3a6214f89e95f7a25618d0f2a1c8e36199c1f8fa59\": rpc error: code = NotFound desc = could not find container \"a2bbc54cb3fbf6aed2870e3a6214f89e95f7a25618d0f2a1c8e36199c1f8fa59\": container with ID starting with a2bbc54cb3fbf6aed2870e3a6214f89e95f7a25618d0f2a1c8e36199c1f8fa59 not found: ID does not exist"
Oct 30 05:14:13.784845 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:13.784821    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh"]
Oct 30 05:14:13.795932 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:13.795913    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-a2f984-fdd8fb47c-x6tlh"]
Oct 30 05:14:14.735432 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:14.735400    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7479a6e1-f414-411e-b1a0-a56fe33f8262" path="/var/lib/kubelet/pods/7479a6e1-f414-411e-b1a0-a56fe33f8262/volumes"
Oct 30 05:14:15.485543 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:15.485508    4874 patch_prober.go:29] interesting pod/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:14:15.485914 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:15.485561    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw" podUID="c4ec0ec0-354e-41f9-897e-ecbd08404b2e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:14:16.127438 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.127378    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 05:14:16.149283 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.149258    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-workload-certs\") pod \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") "
Oct 30 05:14:16.149407 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.149329    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-data\") pod \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") "
Oct 30 05:14:16.149407 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.149384    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-envoy\") pod \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") "
Oct 30 05:14:16.149535 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.149407    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-credential-socket\") pod \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") "
Oct 30 05:14:16.149535 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.149423    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-workload-certs" pod "c4ec0ec0-354e-41f9-897e-ecbd08404b2e" (UID: "c4ec0ec0-354e-41f9-897e-ecbd08404b2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:16.149535 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.149438    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-podinfo\") pod \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") "
Oct 30 05:14:16.149535 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.149457    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-token\" (UniqueName: \"kubernetes.io/projected/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-token\") pod \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") "
Oct 30 05:14:16.149535 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.149477    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-kube-api-access-rckm2\" (UniqueName: \"kubernetes.io/projected/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-kube-api-access-rckm2\") pod \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") "
Oct 30 05:14:16.149772 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.149548    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-workload-socket\") pod \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") "
Oct 30 05:14:16.149772 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.149611    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istiod-ca-cert\") pod \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\" (UID: \"c4ec0ec0-354e-41f9-897e-ecbd08404b2e\") "
Oct 30 05:14:16.149772 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.149621    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-data" pod "c4ec0ec0-354e-41f9-897e-ecbd08404b2e" (UID: "c4ec0ec0-354e-41f9-897e-ecbd08404b2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:16.149772 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.149635    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-credential-socket" pod "c4ec0ec0-354e-41f9-897e-ecbd08404b2e" (UID: "c4ec0ec0-354e-41f9-897e-ecbd08404b2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:16.149983 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.149781    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-workload-socket" pod "c4ec0ec0-354e-41f9-897e-ecbd08404b2e" (UID: "c4ec0ec0-354e-41f9-897e-ecbd08404b2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:16.150048 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.149993    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:16.150048 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.150015    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:16.150048 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.150030    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:16.150048 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.150044    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:16.150220 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.150056    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istiod-ca-cert" pod "c4ec0ec0-354e-41f9-897e-ecbd08404b2e" (UID: "c4ec0ec0-354e-41f9-897e-ecbd08404b2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:14:16.151523 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.151498    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-envoy" pod "c4ec0ec0-354e-41f9-897e-ecbd08404b2e" (UID: "c4ec0ec0-354e-41f9-897e-ecbd08404b2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:16.151692 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.151676    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-kube-api-access-rckm2" pod "c4ec0ec0-354e-41f9-897e-ecbd08404b2e" (UID: "c4ec0ec0-354e-41f9-897e-ecbd08404b2e"). InnerVolumeSpecName "kube-api-access-rckm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:16.152077 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.152056    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-token" pod "c4ec0ec0-354e-41f9-897e-ecbd08404b2e" (UID: "c4ec0ec0-354e-41f9-897e-ecbd08404b2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:16.152077 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.152063    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-podinfo" pod "c4ec0ec0-354e-41f9-897e-ecbd08404b2e" (UID: "c4ec0ec0-354e-41f9-897e-ecbd08404b2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:14:16.250648 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.250617    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:16.250648 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.250645    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:16.250648 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.250653    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:16.250862 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.250661    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:16.250862 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.250670    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rckm2\" (UniqueName: \"kubernetes.io/projected/c4ec0ec0-354e-41f9-897e-ecbd08404b2e-kube-api-access-rckm2\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:16.754156 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.754126    4874 generic.go:356] "Generic (PLEG): container finished" podID="c4ec0ec0-354e-41f9-897e-ecbd08404b2e" containerID="00736f477cb56865fbc045a7404d67a6c67ac5bf01df7e910b151a1b8850e8d2" exitCode=0
Oct 30 05:14:16.754610 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.754170    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"
Oct 30 05:14:16.754610 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.754210    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw" event={"ID":"c4ec0ec0-354e-41f9-897e-ecbd08404b2e","Type":"ContainerDied","Data":"00736f477cb56865fbc045a7404d67a6c67ac5bf01df7e910b151a1b8850e8d2"}
Oct 30 05:14:16.754610 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.754292    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw" event={"ID":"c4ec0ec0-354e-41f9-897e-ecbd08404b2e","Type":"ContainerDied","Data":"dcb431a02afa5c420820c942d20543f51530b2dbae8ab5bc7a6d865761db1b23"}
Oct 30 05:14:16.754610 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.754326    4874 scope.go:122] "RemoveContainer" containerID="00736f477cb56865fbc045a7404d67a6c67ac5bf01df7e910b151a1b8850e8d2"
Oct 30 05:14:16.763873 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.763856    4874 scope.go:122] "RemoveContainer" containerID="67be90f95774f534380bc661b2248bd32e3065be0904800b5eaef4e8f65ac8c9"
Oct 30 05:14:16.772305 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.772289    4874 scope.go:122] "RemoveContainer" containerID="00736f477cb56865fbc045a7404d67a6c67ac5bf01df7e910b151a1b8850e8d2"
Oct 30 05:14:16.772539 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:14:16.772521    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00736f477cb56865fbc045a7404d67a6c67ac5bf01df7e910b151a1b8850e8d2\": container with ID starting with 00736f477cb56865fbc045a7404d67a6c67ac5bf01df7e910b151a1b8850e8d2 not found: ID does not exist" containerID="00736f477cb56865fbc045a7404d67a6c67ac5bf01df7e910b151a1b8850e8d2"
Oct 30 05:14:16.772586 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.772549    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00736f477cb56865fbc045a7404d67a6c67ac5bf01df7e910b151a1b8850e8d2"} err="failed to get container status \"00736f477cb56865fbc045a7404d67a6c67ac5bf01df7e910b151a1b8850e8d2\": rpc error: code = NotFound desc = could not find container \"00736f477cb56865fbc045a7404d67a6c67ac5bf01df7e910b151a1b8850e8d2\": container with ID starting with 00736f477cb56865fbc045a7404d67a6c67ac5bf01df7e910b151a1b8850e8d2 not found: ID does not exist"
Oct 30 05:14:16.772586 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.772568    4874 scope.go:122] "RemoveContainer" containerID="67be90f95774f534380bc661b2248bd32e3065be0904800b5eaef4e8f65ac8c9"
Oct 30 05:14:16.776183 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:14:16.776159    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67be90f95774f534380bc661b2248bd32e3065be0904800b5eaef4e8f65ac8c9\": container with ID starting with 67be90f95774f534380bc661b2248bd32e3065be0904800b5eaef4e8f65ac8c9 not found: ID does not exist" containerID="67be90f95774f534380bc661b2248bd32e3065be0904800b5eaef4e8f65ac8c9"
Oct 30 05:14:16.776292 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.776194    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67be90f95774f534380bc661b2248bd32e3065be0904800b5eaef4e8f65ac8c9"} err="failed to get container status \"67be90f95774f534380bc661b2248bd32e3065be0904800b5eaef4e8f65ac8c9\": rpc error: code = NotFound desc = could not find container \"67be90f95774f534380bc661b2248bd32e3065be0904800b5eaef4e8f65ac8c9\": container with ID starting with 67be90f95774f534380bc661b2248bd32e3065be0904800b5eaef4e8f65ac8c9 not found: ID does not exist"
Oct 30 05:14:16.814937 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.814910    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"]
Oct 30 05:14:16.827008 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:16.826982    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-05bee8-openshift-default-7878c584df-t84sw"]
Oct 30 05:14:18.733143 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:18.733087    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4ec0ec0-354e-41f9-897e-ecbd08404b2e" path="/var/lib/kubelet/pods/c4ec0ec0-354e-41f9-897e-ecbd08404b2e/volumes"
Oct 30 05:14:20.049843 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:20.049806    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn"]
Oct 30 05:14:20.050287 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:20.050045    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn" podUID="20246613-3c91-4fc6-9c96-0e0b81709958" containerName="maas-api" containerID="cri-o://15d09438ac875d56b99bbbdbcbebab7448287b55aa111666f0d0102755082a5e" gracePeriod=30
Oct 30 05:14:20.307526 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:20.307472    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn"
Oct 30 05:14:20.399589 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:20.399554    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20246613-3c91-4fc6-9c96-0e0b81709958-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20246613-3c91-4fc6-9c96-0e0b81709958-openshift-service-ca\") pod \"20246613-3c91-4fc6-9c96-0e0b81709958\" (UID: \"20246613-3c91-4fc6-9c96-0e0b81709958\") "
Oct 30 05:14:20.399742 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:20.399645    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20246613-3c91-4fc6-9c96-0e0b81709958-kube-api-access-fprvf\" (UniqueName: \"kubernetes.io/projected/20246613-3c91-4fc6-9c96-0e0b81709958-kube-api-access-fprvf\") pod \"20246613-3c91-4fc6-9c96-0e0b81709958\" (UID: \"20246613-3c91-4fc6-9c96-0e0b81709958\") "
Oct 30 05:14:20.399808 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:20.399790    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20246613-3c91-4fc6-9c96-0e0b81709958-metrics-certs\" (UniqueName: \"kubernetes.io/secret/20246613-3c91-4fc6-9c96-0e0b81709958-metrics-certs\") pod \"20246613-3c91-4fc6-9c96-0e0b81709958\" (UID: \"20246613-3c91-4fc6-9c96-0e0b81709958\") "
Oct 30 05:14:20.399851 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:20.399810    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20246613-3c91-4fc6-9c96-0e0b81709958-openshift-service-ca" pod "20246613-3c91-4fc6-9c96-0e0b81709958" (UID: "20246613-3c91-4fc6-9c96-0e0b81709958"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:14:20.399892 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:20.399866    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20246613-3c91-4fc6-9c96-0e0b81709958-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20246613-3c91-4fc6-9c96-0e0b81709958-maas-api-tls\") pod \"20246613-3c91-4fc6-9c96-0e0b81709958\" (UID: \"20246613-3c91-4fc6-9c96-0e0b81709958\") "
Oct 30 05:14:20.400160 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:20.400147    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20246613-3c91-4fc6-9c96-0e0b81709958-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:20.401826 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:20.401797    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20246613-3c91-4fc6-9c96-0e0b81709958-metrics-certs" pod "20246613-3c91-4fc6-9c96-0e0b81709958" (UID: "20246613-3c91-4fc6-9c96-0e0b81709958"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:14:20.401925 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:20.401800    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20246613-3c91-4fc6-9c96-0e0b81709958-kube-api-access-fprvf" pod "20246613-3c91-4fc6-9c96-0e0b81709958" (UID: "20246613-3c91-4fc6-9c96-0e0b81709958"). InnerVolumeSpecName "kube-api-access-fprvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:20.401925 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:20.401845    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20246613-3c91-4fc6-9c96-0e0b81709958-maas-api-tls" pod "20246613-3c91-4fc6-9c96-0e0b81709958" (UID: "20246613-3c91-4fc6-9c96-0e0b81709958"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:14:20.501261 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:20.501176    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20246613-3c91-4fc6-9c96-0e0b81709958-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:20.501261 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:20.501221    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20246613-3c91-4fc6-9c96-0e0b81709958-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:20.501261 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:20.501254    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fprvf\" (UniqueName: \"kubernetes.io/projected/20246613-3c91-4fc6-9c96-0e0b81709958-kube-api-access-fprvf\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:20.791892 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:20.791854    4874 generic.go:356] "Generic (PLEG): container finished" podID="20246613-3c91-4fc6-9c96-0e0b81709958" containerID="15d09438ac875d56b99bbbdbcbebab7448287b55aa111666f0d0102755082a5e" exitCode=0
Oct 30 05:14:20.792095 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:20.791909    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn" event={"ID":"20246613-3c91-4fc6-9c96-0e0b81709958","Type":"ContainerDied","Data":"15d09438ac875d56b99bbbdbcbebab7448287b55aa111666f0d0102755082a5e"}
Oct 30 05:14:20.792095 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:20.791956    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn" event={"ID":"20246613-3c91-4fc6-9c96-0e0b81709958","Type":"ContainerDied","Data":"a89b7a18af55f65afa36b09bed6b90be79386fe4963a10b8f8f4e59a047ab466"}
Oct 30 05:14:20.792095 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:20.791985    4874 scope.go:122] "RemoveContainer" containerID="15d09438ac875d56b99bbbdbcbebab7448287b55aa111666f0d0102755082a5e"
Oct 30 05:14:20.792095 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:20.791997    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn"
Oct 30 05:14:20.801036 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:20.801016    4874 scope.go:122] "RemoveContainer" containerID="15d09438ac875d56b99bbbdbcbebab7448287b55aa111666f0d0102755082a5e"
Oct 30 05:14:20.801339 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:14:20.801319    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d09438ac875d56b99bbbdbcbebab7448287b55aa111666f0d0102755082a5e\": container with ID starting with 15d09438ac875d56b99bbbdbcbebab7448287b55aa111666f0d0102755082a5e not found: ID does not exist" containerID="15d09438ac875d56b99bbbdbcbebab7448287b55aa111666f0d0102755082a5e"
Oct 30 05:14:20.801402 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:20.801347    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d09438ac875d56b99bbbdbcbebab7448287b55aa111666f0d0102755082a5e"} err="failed to get container status \"15d09438ac875d56b99bbbdbcbebab7448287b55aa111666f0d0102755082a5e\": rpc error: code = NotFound desc = could not find container \"15d09438ac875d56b99bbbdbcbebab7448287b55aa111666f0d0102755082a5e\": container with ID starting with 15d09438ac875d56b99bbbdbcbebab7448287b55aa111666f0d0102755082a5e not found: ID does not exist"
Oct 30 05:14:20.843950 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:20.843920    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn"]
Oct 30 05:14:20.854575 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:20.854547    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b195d0-78dd57c47d-swwqn"]
Oct 30 05:14:22.637388 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:22.637356    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:14:22.733261 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:22.733204    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20246613-3c91-4fc6-9c96-0e0b81709958" path="/var/lib/kubelet/pods/20246613-3c91-4fc6-9c96-0e0b81709958/volumes"
Oct 30 05:14:24.431072 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:24.431036    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"]
Oct 30 05:14:24.431621 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:24.431325    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr" podUID="f3db5246-707f-4592-8ea0-f09ede62b5af" containerName="istio-proxy" containerID="cri-o://9c4981967ec74204636dad388e0b7127cbd40c2f21ec0f1ca64f8a3d2f6010f7" gracePeriod=30
Oct 30 05:14:25.608821 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:14:25.608784    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:14:26.342468 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:26.342430    4874 patch_prober.go:29] interesting pod/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:14:26.342634 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:26.342500    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr" podUID="f3db5246-707f-4592-8ea0-f09ede62b5af" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:14:26.887933 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:26.887900    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"]
Oct 30 05:14:26.888399 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:26.888162    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l" podUID="c7abcd90-6f30-44af-bdc4-760820351783" containerName="istio-proxy" containerID="cri-o://341e7847bb54be5f574ccbd42de23924ca1fd62481d274d019f0b42d1a3f0bd0" gracePeriod=30
Oct 30 05:14:29.688791 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.688763    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 05:14:29.839812 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.839781    4874 generic.go:356] "Generic (PLEG): container finished" podID="f3db5246-707f-4592-8ea0-f09ede62b5af" containerID="9c4981967ec74204636dad388e0b7127cbd40c2f21ec0f1ca64f8a3d2f6010f7" exitCode=0
Oct 30 05:14:29.839986 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.839826    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"
Oct 30 05:14:29.839986 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.839856    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr" event={"ID":"f3db5246-707f-4592-8ea0-f09ede62b5af","Type":"ContainerDied","Data":"9c4981967ec74204636dad388e0b7127cbd40c2f21ec0f1ca64f8a3d2f6010f7"}
Oct 30 05:14:29.839986 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.839907    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr" event={"ID":"f3db5246-707f-4592-8ea0-f09ede62b5af","Type":"ContainerDied","Data":"c76db20065bfacbd6af05ada1fa0793e3db1253bed3eec6594ecebe3fe607e54"}
Oct 30 05:14:29.839986 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.839935    4874 scope.go:122] "RemoveContainer" containerID="9c4981967ec74204636dad388e0b7127cbd40c2f21ec0f1ca64f8a3d2f6010f7"
Oct 30 05:14:29.846162 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.846141    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-podinfo\") pod \"f3db5246-707f-4592-8ea0-f09ede62b5af\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") "
Oct 30 05:14:29.846301 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.846195    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-token\" (UniqueName: \"kubernetes.io/projected/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-token\") pod \"f3db5246-707f-4592-8ea0-f09ede62b5af\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") "
Oct 30 05:14:29.846301 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.846256    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-workload-socket\") pod \"f3db5246-707f-4592-8ea0-f09ede62b5af\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") "
Oct 30 05:14:29.846428 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.846303    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-workload-certs\") pod \"f3db5246-707f-4592-8ea0-f09ede62b5af\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") "
Oct 30 05:14:29.846428 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.846410    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-credential-socket\") pod \"f3db5246-707f-4592-8ea0-f09ede62b5af\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") "
Oct 30 05:14:29.846537 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.846451    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3db5246-707f-4592-8ea0-f09ede62b5af-kube-api-access-sp8nf\" (UniqueName: \"kubernetes.io/projected/f3db5246-707f-4592-8ea0-f09ede62b5af-kube-api-access-sp8nf\") pod \"f3db5246-707f-4592-8ea0-f09ede62b5af\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") "
Oct 30 05:14:29.846537 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.846477    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-envoy\") pod \"f3db5246-707f-4592-8ea0-f09ede62b5af\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") "
Oct 30 05:14:29.846537 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.846526    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-data\") pod \"f3db5246-707f-4592-8ea0-f09ede62b5af\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") "
Oct 30 05:14:29.846708 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.846524    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-workload-socket" pod "f3db5246-707f-4592-8ea0-f09ede62b5af" (UID: "f3db5246-707f-4592-8ea0-f09ede62b5af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:29.846708 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.846565    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3db5246-707f-4592-8ea0-f09ede62b5af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3db5246-707f-4592-8ea0-f09ede62b5af-istiod-ca-cert\") pod \"f3db5246-707f-4592-8ea0-f09ede62b5af\" (UID: \"f3db5246-707f-4592-8ea0-f09ede62b5af\") "
Oct 30 05:14:29.846708 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.846594    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-workload-certs" pod "f3db5246-707f-4592-8ea0-f09ede62b5af" (UID: "f3db5246-707f-4592-8ea0-f09ede62b5af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:29.846708 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.846660    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-credential-socket" pod "f3db5246-707f-4592-8ea0-f09ede62b5af" (UID: "f3db5246-707f-4592-8ea0-f09ede62b5af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:29.847013 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.846991    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:29.847083 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.847018    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:29.847083 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.847035    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:29.847207 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.847142    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3db5246-707f-4592-8ea0-f09ede62b5af-istiod-ca-cert" pod "f3db5246-707f-4592-8ea0-f09ede62b5af" (UID: "f3db5246-707f-4592-8ea0-f09ede62b5af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:14:29.847384 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.847362    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-data" pod "f3db5246-707f-4592-8ea0-f09ede62b5af" (UID: "f3db5246-707f-4592-8ea0-f09ede62b5af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:29.848837 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.848815    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-token" pod "f3db5246-707f-4592-8ea0-f09ede62b5af" (UID: "f3db5246-707f-4592-8ea0-f09ede62b5af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:29.848951 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.848815    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-podinfo" pod "f3db5246-707f-4592-8ea0-f09ede62b5af" (UID: "f3db5246-707f-4592-8ea0-f09ede62b5af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:14:29.849016 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.848974    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3db5246-707f-4592-8ea0-f09ede62b5af-kube-api-access-sp8nf" pod "f3db5246-707f-4592-8ea0-f09ede62b5af" (UID: "f3db5246-707f-4592-8ea0-f09ede62b5af"). InnerVolumeSpecName "kube-api-access-sp8nf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:29.850047 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.850027    4874 scope.go:122] "RemoveContainer" containerID="9c4981967ec74204636dad388e0b7127cbd40c2f21ec0f1ca64f8a3d2f6010f7"
Oct 30 05:14:29.850356 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.850332    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-envoy" pod "f3db5246-707f-4592-8ea0-f09ede62b5af" (UID: "f3db5246-707f-4592-8ea0-f09ede62b5af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:29.850356 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:14:29.850327    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c4981967ec74204636dad388e0b7127cbd40c2f21ec0f1ca64f8a3d2f6010f7\": container with ID starting with 9c4981967ec74204636dad388e0b7127cbd40c2f21ec0f1ca64f8a3d2f6010f7 not found: ID does not exist" containerID="9c4981967ec74204636dad388e0b7127cbd40c2f21ec0f1ca64f8a3d2f6010f7"
Oct 30 05:14:29.850489 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.850382    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c4981967ec74204636dad388e0b7127cbd40c2f21ec0f1ca64f8a3d2f6010f7"} err="failed to get container status \"9c4981967ec74204636dad388e0b7127cbd40c2f21ec0f1ca64f8a3d2f6010f7\": rpc error: code = NotFound desc = could not find container \"9c4981967ec74204636dad388e0b7127cbd40c2f21ec0f1ca64f8a3d2f6010f7\": container with ID starting with 9c4981967ec74204636dad388e0b7127cbd40c2f21ec0f1ca64f8a3d2f6010f7 not found: ID does not exist"
Oct 30 05:14:29.948529 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.948493    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:29.948529 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.948523    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:29.948529 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.948532    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp8nf\" (UniqueName: \"kubernetes.io/projected/f3db5246-707f-4592-8ea0-f09ede62b5af-kube-api-access-sp8nf\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:29.948835 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.948540    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:29.948835 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.948548    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3db5246-707f-4592-8ea0-f09ede62b5af-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:29.948835 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:29.948557    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3db5246-707f-4592-8ea0-f09ede62b5af-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:30.184905 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:30.184874    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"]
Oct 30 05:14:30.196324 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:30.196292    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-gtvbr"]
Oct 30 05:14:30.728564 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:30.728532    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3db5246-707f-4592-8ea0-f09ede62b5af" path="/var/lib/kubelet/pods/f3db5246-707f-4592-8ea0-f09ede62b5af/volumes"
Oct 30 05:14:32.037936 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.037890    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb"]
Oct 30 05:14:32.084620 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.084589    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz"]
Oct 30 05:14:32.144728 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.144704    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 05:14:32.175972 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.175879    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-credential-socket\") pod \"c7abcd90-6f30-44af-bdc4-760820351783\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") "
Oct 30 05:14:32.175972 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.175925    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7abcd90-6f30-44af-bdc4-760820351783-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7abcd90-6f30-44af-bdc4-760820351783-istio-podinfo\") pod \"c7abcd90-6f30-44af-bdc4-760820351783\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") "
Oct 30 05:14:32.175972 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.175943    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7abcd90-6f30-44af-bdc4-760820351783-kube-api-access-lzb9x\" (UniqueName: \"kubernetes.io/projected/c7abcd90-6f30-44af-bdc4-760820351783-kube-api-access-lzb9x\") pod \"c7abcd90-6f30-44af-bdc4-760820351783\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") "
Oct 30 05:14:32.175972 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.175962    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-workload-socket\") pod \"c7abcd90-6f30-44af-bdc4-760820351783\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") "
Oct 30 05:14:32.176343 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.175991    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-istio-data\") pod \"c7abcd90-6f30-44af-bdc4-760820351783\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") "
Oct 30 05:14:32.176343 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.176048    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7abcd90-6f30-44af-bdc4-760820351783-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7abcd90-6f30-44af-bdc4-760820351783-istiod-ca-cert\") pod \"c7abcd90-6f30-44af-bdc4-760820351783\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") "
Oct 30 05:14:32.176343 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.176081    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-workload-certs\") pod \"c7abcd90-6f30-44af-bdc4-760820351783\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") "
Oct 30 05:14:32.176343 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.176177    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-istio-envoy\") pod \"c7abcd90-6f30-44af-bdc4-760820351783\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") "
Oct 30 05:14:32.176343 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.176224    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7abcd90-6f30-44af-bdc4-760820351783-istio-token\" (UniqueName: \"kubernetes.io/projected/c7abcd90-6f30-44af-bdc4-760820351783-istio-token\") pod \"c7abcd90-6f30-44af-bdc4-760820351783\" (UID: \"c7abcd90-6f30-44af-bdc4-760820351783\") "
Oct 30 05:14:32.176343 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.176257    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-credential-socket" pod "c7abcd90-6f30-44af-bdc4-760820351783" (UID: "c7abcd90-6f30-44af-bdc4-760820351783"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:32.176675 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.176412    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-workload-socket" pod "c7abcd90-6f30-44af-bdc4-760820351783" (UID: "c7abcd90-6f30-44af-bdc4-760820351783"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:32.176675 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.176526    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-workload-certs" pod "c7abcd90-6f30-44af-bdc4-760820351783" (UID: "c7abcd90-6f30-44af-bdc4-760820351783"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:32.176797 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.176682    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:32.176797 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.176700    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:32.176797 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.176714    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:32.177014 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.176941    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-istio-data" pod "c7abcd90-6f30-44af-bdc4-760820351783" (UID: "c7abcd90-6f30-44af-bdc4-760820351783"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:32.177232 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.177174    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7abcd90-6f30-44af-bdc4-760820351783-istiod-ca-cert" pod "c7abcd90-6f30-44af-bdc4-760820351783" (UID: "c7abcd90-6f30-44af-bdc4-760820351783"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:14:32.178382 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.178343    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7abcd90-6f30-44af-bdc4-760820351783-kube-api-access-lzb9x" pod "c7abcd90-6f30-44af-bdc4-760820351783" (UID: "c7abcd90-6f30-44af-bdc4-760820351783"). InnerVolumeSpecName "kube-api-access-lzb9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:32.178647 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.178616    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-istio-envoy" pod "c7abcd90-6f30-44af-bdc4-760820351783" (UID: "c7abcd90-6f30-44af-bdc4-760820351783"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:32.178719 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.178641    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7abcd90-6f30-44af-bdc4-760820351783-istio-podinfo" pod "c7abcd90-6f30-44af-bdc4-760820351783" (UID: "c7abcd90-6f30-44af-bdc4-760820351783"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:14:32.178833 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.178818    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7abcd90-6f30-44af-bdc4-760820351783-istio-token" pod "c7abcd90-6f30-44af-bdc4-760820351783" (UID: "c7abcd90-6f30-44af-bdc4-760820351783"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:32.283264 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.279993    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7abcd90-6f30-44af-bdc4-760820351783-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:32.283264 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.280035    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzb9x\" (UniqueName: \"kubernetes.io/projected/c7abcd90-6f30-44af-bdc4-760820351783-kube-api-access-lzb9x\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:32.283264 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.280052    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:32.283264 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.280067    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7abcd90-6f30-44af-bdc4-760820351783-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:32.283264 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.280080    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7abcd90-6f30-44af-bdc4-760820351783-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:32.283264 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.280092    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7abcd90-6f30-44af-bdc4-760820351783-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:32.856142 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.856107    4874 generic.go:356] "Generic (PLEG): container finished" podID="c7abcd90-6f30-44af-bdc4-760820351783" containerID="341e7847bb54be5f574ccbd42de23924ca1fd62481d274d019f0b42d1a3f0bd0" exitCode=0
Oct 30 05:14:32.856358 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.856154    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"
Oct 30 05:14:32.856358 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.856185    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l" event={"ID":"c7abcd90-6f30-44af-bdc4-760820351783","Type":"ContainerDied","Data":"341e7847bb54be5f574ccbd42de23924ca1fd62481d274d019f0b42d1a3f0bd0"}
Oct 30 05:14:32.856358 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.856246    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l" event={"ID":"c7abcd90-6f30-44af-bdc4-760820351783","Type":"ContainerDied","Data":"e3ec4c115e9010847ea5d9df56336d053119445eb41cc30c4184388bd6bd8d3a"}
Oct 30 05:14:32.856358 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.856259    4874 scope.go:122] "RemoveContainer" containerID="341e7847bb54be5f574ccbd42de23924ca1fd62481d274d019f0b42d1a3f0bd0"
Oct 30 05:14:32.865462 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.865440    4874 scope.go:122] "RemoveContainer" containerID="050398fb1f680c4957e138b470dc3a80d22c3535671abfae8715fb6ce1a29536"
Oct 30 05:14:32.873820 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.873798    4874 scope.go:122] "RemoveContainer" containerID="341e7847bb54be5f574ccbd42de23924ca1fd62481d274d019f0b42d1a3f0bd0"
Oct 30 05:14:32.874056 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:14:32.874036    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"341e7847bb54be5f574ccbd42de23924ca1fd62481d274d019f0b42d1a3f0bd0\": container with ID starting with 341e7847bb54be5f574ccbd42de23924ca1fd62481d274d019f0b42d1a3f0bd0 not found: ID does not exist" containerID="341e7847bb54be5f574ccbd42de23924ca1fd62481d274d019f0b42d1a3f0bd0"
Oct 30 05:14:32.874107 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.874064    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"341e7847bb54be5f574ccbd42de23924ca1fd62481d274d019f0b42d1a3f0bd0"} err="failed to get container status \"341e7847bb54be5f574ccbd42de23924ca1fd62481d274d019f0b42d1a3f0bd0\": rpc error: code = NotFound desc = could not find container \"341e7847bb54be5f574ccbd42de23924ca1fd62481d274d019f0b42d1a3f0bd0\": container with ID starting with 341e7847bb54be5f574ccbd42de23924ca1fd62481d274d019f0b42d1a3f0bd0 not found: ID does not exist"
Oct 30 05:14:32.874107 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.874082    4874 scope.go:122] "RemoveContainer" containerID="050398fb1f680c4957e138b470dc3a80d22c3535671abfae8715fb6ce1a29536"
Oct 30 05:14:32.874330 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:14:32.874311    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"050398fb1f680c4957e138b470dc3a80d22c3535671abfae8715fb6ce1a29536\": container with ID starting with 050398fb1f680c4957e138b470dc3a80d22c3535671abfae8715fb6ce1a29536 not found: ID does not exist" containerID="050398fb1f680c4957e138b470dc3a80d22c3535671abfae8715fb6ce1a29536"
Oct 30 05:14:32.874389 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.874340    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"050398fb1f680c4957e138b470dc3a80d22c3535671abfae8715fb6ce1a29536"} err="failed to get container status \"050398fb1f680c4957e138b470dc3a80d22c3535671abfae8715fb6ce1a29536\": rpc error: code = NotFound desc = could not find container \"050398fb1f680c4957e138b470dc3a80d22c3535671abfae8715fb6ce1a29536\": container with ID starting with 050398fb1f680c4957e138b470dc3a80d22c3535671abfae8715fb6ce1a29536 not found: ID does not exist"
Oct 30 05:14:32.915144 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.915123    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"]
Oct 30 05:14:32.926542 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:32.926518    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-b195d0-openshift-default-6647b697d5-twx2l"]
Oct 30 05:14:33.796911 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:33.796875    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx"]
Oct 30 05:14:34.728009 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:34.727976    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7abcd90-6f30-44af-bdc4-760820351783" path="/var/lib/kubelet/pods/c7abcd90-6f30-44af-bdc4-760820351783/volumes"
Oct 30 05:14:37.038951 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:37.038904    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb" podUID="653f8461-4436-4506-bc3e-0271173d425e" containerName="payload-processing" containerID="cri-o://5be2d8adc28c67f08f9e0ec56639112b0f04c4e4cb9132c4c8c85e30ac1d33fb" gracePeriod=25
Oct 30 05:14:37.085107 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:37.085048    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz" podUID="beb04a4a-dc5c-4583-a464-3c56374faa33" containerName="payload-pre-processing" containerID="cri-o://cd59cb01d7f88ed626c28acef56532c26568abd7ab4c04189f30a91ec706e324" gracePeriod=25
Oct 30 05:14:37.372123 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:37.372036    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz" podUID="beb04a4a-dc5c-4583-a464-3c56374faa33" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:14:37.372287 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:37.372041    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb" podUID="653f8461-4436-4506-bc3e-0271173d425e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:14:38.797718 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:38.797664    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx" podUID="9be3939f-afda-4003-bd81-55e3349fdd86" containerName="payload-pre-processing" containerID="cri-o://238f5d8804dd5366de753024615f9b4783e452d8cbbf983397116663c3e742ce" gracePeriod=25
Oct 30 05:14:39.171347 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:39.171260    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:14:46.277917 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:46.277876    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj"]
Oct 30 05:14:47.371902 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:47.371866    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb" podUID="653f8461-4436-4506-bc3e-0271173d425e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:14:47.372321 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:47.371949    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz" podUID="beb04a4a-dc5c-4583-a464-3c56374faa33" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:14:48.311822 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:48.311780    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx" podUID="9be3939f-afda-4003-bd81-55e3349fdd86" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:14:48.694322 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:48.694218    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"]
Oct 30 05:14:48.694797 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:48.694444    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs" podUID="833c472c-0226-4e47-bc11-5d2659845a86" containerName="istio-proxy" containerID="cri-o://35c276127baf4f367c64e15b4614c9c08c0d99686ea82bceded1649a17163ee4" gracePeriod=30
Oct 30 05:14:49.644455 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:49.644418    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:14:51.279685 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:51.279622    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj" podUID="6e900ea6-5bdb-40c7-bcd2-de0a6910aac0" containerName="payload-processing" containerID="cri-o://6a93dc9501e984d7b9dfbb1b22493e35ce6fb6da3432a1e1e502c0d12deab77e" gracePeriod=25
Oct 30 05:14:52.723650 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:52.723612    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj" podUID="6e900ea6-5bdb-40c7-bcd2-de0a6910aac0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:14:53.937539 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:53.937515    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 05:14:53.996525 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:53.996492    4874 generic.go:356] "Generic (PLEG): container finished" podID="833c472c-0226-4e47-bc11-5d2659845a86" containerID="35c276127baf4f367c64e15b4614c9c08c0d99686ea82bceded1649a17163ee4" exitCode=0
Oct 30 05:14:53.996691 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:53.996548    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"
Oct 30 05:14:53.996691 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:53.996581    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs" event={"ID":"833c472c-0226-4e47-bc11-5d2659845a86","Type":"ContainerDied","Data":"35c276127baf4f367c64e15b4614c9c08c0d99686ea82bceded1649a17163ee4"}
Oct 30 05:14:53.996691 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:53.996639    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs" event={"ID":"833c472c-0226-4e47-bc11-5d2659845a86","Type":"ContainerDied","Data":"cd2f2e6203a64b3056232925dbfb780bbc43535e907e169d361e86666d9f9fd5"}
Oct 30 05:14:53.996691 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:53.996671    4874 scope.go:122] "RemoveContainer" containerID="35c276127baf4f367c64e15b4614c9c08c0d99686ea82bceded1649a17163ee4"
Oct 30 05:14:54.007350 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.007316    4874 scope.go:122] "RemoveContainer" containerID="ca61199c97394dfa8d2e03da7988c0080ccbbee9721005442a3ca2dbe62eb7c8"
Oct 30 05:14:54.015304 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.015275    4874 scope.go:122] "RemoveContainer" containerID="35c276127baf4f367c64e15b4614c9c08c0d99686ea82bceded1649a17163ee4"
Oct 30 05:14:54.015583 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:14:54.015557    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35c276127baf4f367c64e15b4614c9c08c0d99686ea82bceded1649a17163ee4\": container with ID starting with 35c276127baf4f367c64e15b4614c9c08c0d99686ea82bceded1649a17163ee4 not found: ID does not exist" containerID="35c276127baf4f367c64e15b4614c9c08c0d99686ea82bceded1649a17163ee4"
Oct 30 05:14:54.015674 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.015594    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35c276127baf4f367c64e15b4614c9c08c0d99686ea82bceded1649a17163ee4"} err="failed to get container status \"35c276127baf4f367c64e15b4614c9c08c0d99686ea82bceded1649a17163ee4\": rpc error: code = NotFound desc = could not find container \"35c276127baf4f367c64e15b4614c9c08c0d99686ea82bceded1649a17163ee4\": container with ID starting with 35c276127baf4f367c64e15b4614c9c08c0d99686ea82bceded1649a17163ee4 not found: ID does not exist"
Oct 30 05:14:54.015674 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.015619    4874 scope.go:122] "RemoveContainer" containerID="ca61199c97394dfa8d2e03da7988c0080ccbbee9721005442a3ca2dbe62eb7c8"
Oct 30 05:14:54.015915 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:14:54.015894    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca61199c97394dfa8d2e03da7988c0080ccbbee9721005442a3ca2dbe62eb7c8\": container with ID starting with ca61199c97394dfa8d2e03da7988c0080ccbbee9721005442a3ca2dbe62eb7c8 not found: ID does not exist" containerID="ca61199c97394dfa8d2e03da7988c0080ccbbee9721005442a3ca2dbe62eb7c8"
Oct 30 05:14:54.015970 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.015922    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca61199c97394dfa8d2e03da7988c0080ccbbee9721005442a3ca2dbe62eb7c8"} err="failed to get container status \"ca61199c97394dfa8d2e03da7988c0080ccbbee9721005442a3ca2dbe62eb7c8\": rpc error: code = NotFound desc = could not find container \"ca61199c97394dfa8d2e03da7988c0080ccbbee9721005442a3ca2dbe62eb7c8\": container with ID starting with ca61199c97394dfa8d2e03da7988c0080ccbbee9721005442a3ca2dbe62eb7c8 not found: ID does not exist"
Oct 30 05:14:54.017146 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.017125    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/833c472c-0226-4e47-bc11-5d2659845a86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/833c472c-0226-4e47-bc11-5d2659845a86-istiod-ca-cert\") pod \"833c472c-0226-4e47-bc11-5d2659845a86\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") "
Oct 30 05:14:54.017293 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.017226    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/833c472c-0226-4e47-bc11-5d2659845a86-kube-api-access-j9cgw\" (UniqueName: \"kubernetes.io/projected/833c472c-0226-4e47-bc11-5d2659845a86-kube-api-access-j9cgw\") pod \"833c472c-0226-4e47-bc11-5d2659845a86\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") "
Oct 30 05:14:54.017424 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.017394    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/833c472c-0226-4e47-bc11-5d2659845a86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/833c472c-0226-4e47-bc11-5d2659845a86-istio-podinfo\") pod \"833c472c-0226-4e47-bc11-5d2659845a86\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") "
Oct 30 05:14:54.017494 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.017459    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/833c472c-0226-4e47-bc11-5d2659845a86-istio-token\" (UniqueName: \"kubernetes.io/projected/833c472c-0226-4e47-bc11-5d2659845a86-istio-token\") pod \"833c472c-0226-4e47-bc11-5d2659845a86\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") "
Oct 30 05:14:54.017557 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.017496    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-workload-socket\") pod \"833c472c-0226-4e47-bc11-5d2659845a86\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") "
Oct 30 05:14:54.017557 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.017460    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/833c472c-0226-4e47-bc11-5d2659845a86-istiod-ca-cert" pod "833c472c-0226-4e47-bc11-5d2659845a86" (UID: "833c472c-0226-4e47-bc11-5d2659845a86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:14:54.017557 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.017539    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-istio-envoy\") pod \"833c472c-0226-4e47-bc11-5d2659845a86\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") "
Oct 30 05:14:54.017728 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.017618    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-credential-socket\") pod \"833c472c-0226-4e47-bc11-5d2659845a86\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") "
Oct 30 05:14:54.017728 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.017656    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-workload-certs\") pod \"833c472c-0226-4e47-bc11-5d2659845a86\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") "
Oct 30 05:14:54.017728 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.017697    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-istio-data\") pod \"833c472c-0226-4e47-bc11-5d2659845a86\" (UID: \"833c472c-0226-4e47-bc11-5d2659845a86\") "
Oct 30 05:14:54.017896 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.017818    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-workload-socket" pod "833c472c-0226-4e47-bc11-5d2659845a86" (UID: "833c472c-0226-4e47-bc11-5d2659845a86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:54.017959 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.017932    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-credential-socket" pod "833c472c-0226-4e47-bc11-5d2659845a86" (UID: "833c472c-0226-4e47-bc11-5d2659845a86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:54.018040 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.018021    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-istio-data" pod "833c472c-0226-4e47-bc11-5d2659845a86" (UID: "833c472c-0226-4e47-bc11-5d2659845a86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:54.018157 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.018129    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-workload-certs" pod "833c472c-0226-4e47-bc11-5d2659845a86" (UID: "833c472c-0226-4e47-bc11-5d2659845a86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:54.018257 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.018204    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:54.018257 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.018225    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:54.018390 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.018256    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:54.018390 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.018277    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/833c472c-0226-4e47-bc11-5d2659845a86-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:54.019711 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.019683    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833c472c-0226-4e47-bc11-5d2659845a86-istio-token" pod "833c472c-0226-4e47-bc11-5d2659845a86" (UID: "833c472c-0226-4e47-bc11-5d2659845a86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:54.019982 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.019958    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-istio-envoy" pod "833c472c-0226-4e47-bc11-5d2659845a86" (UID: "833c472c-0226-4e47-bc11-5d2659845a86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:14:54.020153 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.020135    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/833c472c-0226-4e47-bc11-5d2659845a86-istio-podinfo" pod "833c472c-0226-4e47-bc11-5d2659845a86" (UID: "833c472c-0226-4e47-bc11-5d2659845a86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:14:54.020272 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.020254    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833c472c-0226-4e47-bc11-5d2659845a86-kube-api-access-j9cgw" pod "833c472c-0226-4e47-bc11-5d2659845a86" (UID: "833c472c-0226-4e47-bc11-5d2659845a86"). InnerVolumeSpecName "kube-api-access-j9cgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:54.119415 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.119388    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9cgw\" (UniqueName: \"kubernetes.io/projected/833c472c-0226-4e47-bc11-5d2659845a86-kube-api-access-j9cgw\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:54.119415 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.119411    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/833c472c-0226-4e47-bc11-5d2659845a86-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:54.119415 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.119420    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/833c472c-0226-4e47-bc11-5d2659845a86-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:54.119627 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.119429    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:54.119627 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.119436    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/833c472c-0226-4e47-bc11-5d2659845a86-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:54.340849 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.340819    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"]
Oct 30 05:14:54.346049 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.346020    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-mzpcs"]
Oct 30 05:14:54.437395 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.437360    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27"]
Oct 30 05:14:54.437657 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.437623    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27" podUID="14f989b1-ee76-4cef-950c-ed97b8a5b19b" containerName="maas-api" containerID="cri-o://0a136b637e470720fad8a50c08f21e65d24e76b05fe51f3753dc09eb46939691" gracePeriod=30
Oct 30 05:14:54.678154 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.678122    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27"
Oct 30 05:14:54.723207 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.723180    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14f989b1-ee76-4cef-950c-ed97b8a5b19b-kube-api-access-m4bl5\" (UniqueName: \"kubernetes.io/projected/14f989b1-ee76-4cef-950c-ed97b8a5b19b-kube-api-access-m4bl5\") pod \"14f989b1-ee76-4cef-950c-ed97b8a5b19b\" (UID: \"14f989b1-ee76-4cef-950c-ed97b8a5b19b\") "
Oct 30 05:14:54.723383 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.723228    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14f989b1-ee76-4cef-950c-ed97b8a5b19b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/14f989b1-ee76-4cef-950c-ed97b8a5b19b-metrics-certs\") pod \"14f989b1-ee76-4cef-950c-ed97b8a5b19b\" (UID: \"14f989b1-ee76-4cef-950c-ed97b8a5b19b\") "
Oct 30 05:14:54.723383 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.723271    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14f989b1-ee76-4cef-950c-ed97b8a5b19b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14f989b1-ee76-4cef-950c-ed97b8a5b19b-maas-api-tls\") pod \"14f989b1-ee76-4cef-950c-ed97b8a5b19b\" (UID: \"14f989b1-ee76-4cef-950c-ed97b8a5b19b\") "
Oct 30 05:14:54.723533 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.723394    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14f989b1-ee76-4cef-950c-ed97b8a5b19b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14f989b1-ee76-4cef-950c-ed97b8a5b19b-openshift-service-ca\") pod \"14f989b1-ee76-4cef-950c-ed97b8a5b19b\" (UID: \"14f989b1-ee76-4cef-950c-ed97b8a5b19b\") "
Oct 30 05:14:54.723875 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.723845    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14f989b1-ee76-4cef-950c-ed97b8a5b19b-openshift-service-ca" pod "14f989b1-ee76-4cef-950c-ed97b8a5b19b" (UID: "14f989b1-ee76-4cef-950c-ed97b8a5b19b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:14:54.726057 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.726033    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14f989b1-ee76-4cef-950c-ed97b8a5b19b-metrics-certs" pod "14f989b1-ee76-4cef-950c-ed97b8a5b19b" (UID: "14f989b1-ee76-4cef-950c-ed97b8a5b19b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:14:54.726057 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.726044    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f989b1-ee76-4cef-950c-ed97b8a5b19b-kube-api-access-m4bl5" pod "14f989b1-ee76-4cef-950c-ed97b8a5b19b" (UID: "14f989b1-ee76-4cef-950c-ed97b8a5b19b"). InnerVolumeSpecName "kube-api-access-m4bl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:54.726345 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.726324    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14f989b1-ee76-4cef-950c-ed97b8a5b19b-maas-api-tls" pod "14f989b1-ee76-4cef-950c-ed97b8a5b19b" (UID: "14f989b1-ee76-4cef-950c-ed97b8a5b19b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:14:54.730368 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.730336    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="833c472c-0226-4e47-bc11-5d2659845a86" path="/var/lib/kubelet/pods/833c472c-0226-4e47-bc11-5d2659845a86/volumes"
Oct 30 05:14:54.824730 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.824702    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14f989b1-ee76-4cef-950c-ed97b8a5b19b-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:54.824730 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.824729    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4bl5\" (UniqueName: \"kubernetes.io/projected/14f989b1-ee76-4cef-950c-ed97b8a5b19b-kube-api-access-m4bl5\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:54.824884 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.824744    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14f989b1-ee76-4cef-950c-ed97b8a5b19b-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:54.824884 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:54.824755    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14f989b1-ee76-4cef-950c-ed97b8a5b19b-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:55.004182 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:55.004147    4874 generic.go:356] "Generic (PLEG): container finished" podID="14f989b1-ee76-4cef-950c-ed97b8a5b19b" containerID="0a136b637e470720fad8a50c08f21e65d24e76b05fe51f3753dc09eb46939691" exitCode=0
Oct 30 05:14:55.004671 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:55.004189    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27"
Oct 30 05:14:55.004671 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:55.004258    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27" event={"ID":"14f989b1-ee76-4cef-950c-ed97b8a5b19b","Type":"ContainerDied","Data":"0a136b637e470720fad8a50c08f21e65d24e76b05fe51f3753dc09eb46939691"}
Oct 30 05:14:55.004671 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:55.004322    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27" event={"ID":"14f989b1-ee76-4cef-950c-ed97b8a5b19b","Type":"ContainerDied","Data":"b505d9c39e73d94bf1d9054420b1757b554e1b086907723739a80eb4abdb343e"}
Oct 30 05:14:55.004671 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:55.004356    4874 scope.go:122] "RemoveContainer" containerID="0a136b637e470720fad8a50c08f21e65d24e76b05fe51f3753dc09eb46939691"
Oct 30 05:14:55.013293 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:55.013265    4874 scope.go:122] "RemoveContainer" containerID="0a136b637e470720fad8a50c08f21e65d24e76b05fe51f3753dc09eb46939691"
Oct 30 05:14:55.013572 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:14:55.013553    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a136b637e470720fad8a50c08f21e65d24e76b05fe51f3753dc09eb46939691\": container with ID starting with 0a136b637e470720fad8a50c08f21e65d24e76b05fe51f3753dc09eb46939691 not found: ID does not exist" containerID="0a136b637e470720fad8a50c08f21e65d24e76b05fe51f3753dc09eb46939691"
Oct 30 05:14:55.013628 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:55.013580    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a136b637e470720fad8a50c08f21e65d24e76b05fe51f3753dc09eb46939691"} err="failed to get container status \"0a136b637e470720fad8a50c08f21e65d24e76b05fe51f3753dc09eb46939691\": rpc error: code = NotFound desc = could not find container \"0a136b637e470720fad8a50c08f21e65d24e76b05fe51f3753dc09eb46939691\": container with ID starting with 0a136b637e470720fad8a50c08f21e65d24e76b05fe51f3753dc09eb46939691 not found: ID does not exist"
Oct 30 05:14:55.052467 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:55.052437    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27"]
Oct 30 05:14:55.054742 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:55.054716    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-247158-5fbdcd6764-qhh27"]
Oct 30 05:14:55.608990 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:14:55.608955    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:14:56.727676 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:56.727641    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14f989b1-ee76-4cef-950c-ed97b8a5b19b" path="/var/lib/kubelet/pods/14f989b1-ee76-4cef-950c-ed97b8a5b19b/volumes"
Oct 30 05:14:57.353850 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:57.353824    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb"
Oct 30 05:14:57.376878 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:57.376855    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz"
Oct 30 05:14:57.448675 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:57.448634    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/653f8461-4436-4506-bc3e-0271173d425e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653f8461-4436-4506-bc3e-0271173d425e-plugins-config-volume\") pod \"653f8461-4436-4506-bc3e-0271173d425e\" (UID: \"653f8461-4436-4506-bc3e-0271173d425e\") "
Oct 30 05:14:57.448886 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:57.448713    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beb04a4a-dc5c-4583-a464-3c56374faa33-kube-api-access-cwp8d\" (UniqueName: \"kubernetes.io/projected/beb04a4a-dc5c-4583-a464-3c56374faa33-kube-api-access-cwp8d\") pod \"beb04a4a-dc5c-4583-a464-3c56374faa33\" (UID: \"beb04a4a-dc5c-4583-a464-3c56374faa33\") "
Oct 30 05:14:57.448886 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:57.448791    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/653f8461-4436-4506-bc3e-0271173d425e-kube-api-access-dd7z6\" (UniqueName: \"kubernetes.io/projected/653f8461-4436-4506-bc3e-0271173d425e-kube-api-access-dd7z6\") pod \"653f8461-4436-4506-bc3e-0271173d425e\" (UID: \"653f8461-4436-4506-bc3e-0271173d425e\") "
Oct 30 05:14:57.449021 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:57.448905    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beb04a4a-dc5c-4583-a464-3c56374faa33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/beb04a4a-dc5c-4583-a464-3c56374faa33-plugins-config-volume\") pod \"beb04a4a-dc5c-4583-a464-3c56374faa33\" (UID: \"beb04a4a-dc5c-4583-a464-3c56374faa33\") "
Oct 30 05:14:57.449108 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:57.449080    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/653f8461-4436-4506-bc3e-0271173d425e-plugins-config-volume" pod "653f8461-4436-4506-bc3e-0271173d425e" (UID: "653f8461-4436-4506-bc3e-0271173d425e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:14:57.449397 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:57.449274    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653f8461-4436-4506-bc3e-0271173d425e-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:57.449915 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:57.449875    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beb04a4a-dc5c-4583-a464-3c56374faa33-plugins-config-volume" pod "beb04a4a-dc5c-4583-a464-3c56374faa33" (UID: "beb04a4a-dc5c-4583-a464-3c56374faa33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:14:57.452962 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:57.452650    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb04a4a-dc5c-4583-a464-3c56374faa33-kube-api-access-cwp8d" pod "beb04a4a-dc5c-4583-a464-3c56374faa33" (UID: "beb04a4a-dc5c-4583-a464-3c56374faa33"). InnerVolumeSpecName "kube-api-access-cwp8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:57.452962 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:57.452842    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653f8461-4436-4506-bc3e-0271173d425e-kube-api-access-dd7z6" pod "653f8461-4436-4506-bc3e-0271173d425e" (UID: "653f8461-4436-4506-bc3e-0271173d425e"). InnerVolumeSpecName "kube-api-access-dd7z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:57.550723 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:57.550651    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd7z6\" (UniqueName: \"kubernetes.io/projected/653f8461-4436-4506-bc3e-0271173d425e-kube-api-access-dd7z6\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:57.550723 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:57.550678    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/beb04a4a-dc5c-4583-a464-3c56374faa33-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:57.550723 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:57.550688    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwp8d\" (UniqueName: \"kubernetes.io/projected/beb04a4a-dc5c-4583-a464-3c56374faa33-kube-api-access-cwp8d\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:58.022979 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:58.022947    4874 generic.go:356] "Generic (PLEG): container finished" podID="653f8461-4436-4506-bc3e-0271173d425e" containerID="5be2d8adc28c67f08f9e0ec56639112b0f04c4e4cb9132c4c8c85e30ac1d33fb" exitCode=0
Oct 30 05:14:58.023443 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:58.022987    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb"
Oct 30 05:14:58.023443 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:58.023028    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb" event={"ID":"653f8461-4436-4506-bc3e-0271173d425e","Type":"ContainerDied","Data":"5be2d8adc28c67f08f9e0ec56639112b0f04c4e4cb9132c4c8c85e30ac1d33fb"}
Oct 30 05:14:58.023443 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:58.023078    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb" event={"ID":"653f8461-4436-4506-bc3e-0271173d425e","Type":"ContainerDied","Data":"3c5b9226c26d660050a4c640c3d72ece18f27a916e53c23b54bca3581b2675f0"}
Oct 30 05:14:58.023443 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:58.023105    4874 scope.go:122] "RemoveContainer" containerID="5be2d8adc28c67f08f9e0ec56639112b0f04c4e4cb9132c4c8c85e30ac1d33fb"
Oct 30 05:14:58.024486 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:58.024466    4874 generic.go:356] "Generic (PLEG): container finished" podID="beb04a4a-dc5c-4583-a464-3c56374faa33" containerID="cd59cb01d7f88ed626c28acef56532c26568abd7ab4c04189f30a91ec706e324" exitCode=0
Oct 30 05:14:58.024599 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:58.024514    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz" event={"ID":"beb04a4a-dc5c-4583-a464-3c56374faa33","Type":"ContainerDied","Data":"cd59cb01d7f88ed626c28acef56532c26568abd7ab4c04189f30a91ec706e324"}
Oct 30 05:14:58.024599 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:58.024518    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz"
Oct 30 05:14:58.024599 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:58.024552    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz" event={"ID":"beb04a4a-dc5c-4583-a464-3c56374faa33","Type":"ContainerDied","Data":"fc1a3388d5a5e2dc3c9701d9213386ccd230723db7c5a03da72bc1a69b7fc939"}
Oct 30 05:14:58.032655 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:58.032637    4874 scope.go:122] "RemoveContainer" containerID="5be2d8adc28c67f08f9e0ec56639112b0f04c4e4cb9132c4c8c85e30ac1d33fb"
Oct 30 05:14:58.032915 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:14:58.032895    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5be2d8adc28c67f08f9e0ec56639112b0f04c4e4cb9132c4c8c85e30ac1d33fb\": container with ID starting with 5be2d8adc28c67f08f9e0ec56639112b0f04c4e4cb9132c4c8c85e30ac1d33fb not found: ID does not exist" containerID="5be2d8adc28c67f08f9e0ec56639112b0f04c4e4cb9132c4c8c85e30ac1d33fb"
Oct 30 05:14:58.033011 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:58.032927    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5be2d8adc28c67f08f9e0ec56639112b0f04c4e4cb9132c4c8c85e30ac1d33fb"} err="failed to get container status \"5be2d8adc28c67f08f9e0ec56639112b0f04c4e4cb9132c4c8c85e30ac1d33fb\": rpc error: code = NotFound desc = could not find container \"5be2d8adc28c67f08f9e0ec56639112b0f04c4e4cb9132c4c8c85e30ac1d33fb\": container with ID starting with 5be2d8adc28c67f08f9e0ec56639112b0f04c4e4cb9132c4c8c85e30ac1d33fb not found: ID does not exist"
Oct 30 05:14:58.033011 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:58.032949    4874 scope.go:122] "RemoveContainer" containerID="cd59cb01d7f88ed626c28acef56532c26568abd7ab4c04189f30a91ec706e324"
Oct 30 05:14:58.040649 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:58.040629    4874 scope.go:122] "RemoveContainer" containerID="cd59cb01d7f88ed626c28acef56532c26568abd7ab4c04189f30a91ec706e324"
Oct 30 05:14:58.040897 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:14:58.040877    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd59cb01d7f88ed626c28acef56532c26568abd7ab4c04189f30a91ec706e324\": container with ID starting with cd59cb01d7f88ed626c28acef56532c26568abd7ab4c04189f30a91ec706e324 not found: ID does not exist" containerID="cd59cb01d7f88ed626c28acef56532c26568abd7ab4c04189f30a91ec706e324"
Oct 30 05:14:58.040940 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:58.040905    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd59cb01d7f88ed626c28acef56532c26568abd7ab4c04189f30a91ec706e324"} err="failed to get container status \"cd59cb01d7f88ed626c28acef56532c26568abd7ab4c04189f30a91ec706e324\": rpc error: code = NotFound desc = could not find container \"cd59cb01d7f88ed626c28acef56532c26568abd7ab4c04189f30a91ec706e324\": container with ID starting with cd59cb01d7f88ed626c28acef56532c26568abd7ab4c04189f30a91ec706e324 not found: ID does not exist"
Oct 30 05:14:58.071212 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:58.071191    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz"]
Oct 30 05:14:58.082780 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:58.082761    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz"]
Oct 30 05:14:58.107953 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:58.107932    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb"]
Oct 30 05:14:58.111607 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:58.111586    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e6b837-5b7557b97d-7wsdb"]
Oct 30 05:14:58.311768 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:58.311690    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx" podUID="9be3939f-afda-4003-bd81-55e3349fdd86" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:14:58.370747 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:58.370715    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e6b837-6f8d855b9f-xlfcz" podUID="beb04a4a-dc5c-4583-a464-3c56374faa33" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.124:50052\" within 1s: context deadline exceeded"
Oct 30 05:14:58.370871 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:14:58.370800    4874 logging.go:55] [core] [Channel #7133 SubChannel #7134]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.124:50052", ServerName: "10.134.0.124:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.124:50052: operation was canceled"
Oct 30 05:14:58.728124 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:58.728098    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="653f8461-4436-4506-bc3e-0271173d425e" path="/var/lib/kubelet/pods/653f8461-4436-4506-bc3e-0271173d425e/volumes"
Oct 30 05:14:58.728532 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:58.728518    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beb04a4a-dc5c-4583-a464-3c56374faa33" path="/var/lib/kubelet/pods/beb04a4a-dc5c-4583-a464-3c56374faa33/volumes"
Oct 30 05:14:59.032415 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:59.032386    4874 generic.go:356] "Generic (PLEG): container finished" podID="9be3939f-afda-4003-bd81-55e3349fdd86" containerID="238f5d8804dd5366de753024615f9b4783e452d8cbbf983397116663c3e742ce" exitCode=0
Oct 30 05:14:59.032817 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:59.032474    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx" event={"ID":"9be3939f-afda-4003-bd81-55e3349fdd86","Type":"ContainerDied","Data":"238f5d8804dd5366de753024615f9b4783e452d8cbbf983397116663c3e742ce"}
Oct 30 05:14:59.047426 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:59.047408    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx"
Oct 30 05:14:59.169517 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:59.169492    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9be3939f-afda-4003-bd81-55e3349fdd86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9be3939f-afda-4003-bd81-55e3349fdd86-plugins-config-volume\") pod \"9be3939f-afda-4003-bd81-55e3349fdd86\" (UID: \"9be3939f-afda-4003-bd81-55e3349fdd86\") "
Oct 30 05:14:59.169648 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:59.169583    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9be3939f-afda-4003-bd81-55e3349fdd86-kube-api-access-4fjk5\" (UniqueName: \"kubernetes.io/projected/9be3939f-afda-4003-bd81-55e3349fdd86-kube-api-access-4fjk5\") pod \"9be3939f-afda-4003-bd81-55e3349fdd86\" (UID: \"9be3939f-afda-4003-bd81-55e3349fdd86\") "
Oct 30 05:14:59.169925 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:59.169901    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9be3939f-afda-4003-bd81-55e3349fdd86-plugins-config-volume" pod "9be3939f-afda-4003-bd81-55e3349fdd86" (UID: "9be3939f-afda-4003-bd81-55e3349fdd86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:14:59.171894 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:59.171865    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be3939f-afda-4003-bd81-55e3349fdd86-kube-api-access-4fjk5" pod "9be3939f-afda-4003-bd81-55e3349fdd86" (UID: "9be3939f-afda-4003-bd81-55e3349fdd86"). InnerVolumeSpecName "kube-api-access-4fjk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:14:59.270701 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:59.270677    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fjk5\" (UniqueName: \"kubernetes.io/projected/9be3939f-afda-4003-bd81-55e3349fdd86-kube-api-access-4fjk5\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:59.270701 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:59.270700    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9be3939f-afda-4003-bd81-55e3349fdd86-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:14:59.625274 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:14:59.611413    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:15:00.039689 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:00.039661    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx"
Oct 30 05:15:00.040147 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:00.039696    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx" event={"ID":"9be3939f-afda-4003-bd81-55e3349fdd86","Type":"ContainerDied","Data":"befe2d885adab999e3332b6f9d9daac492f2f93a2435f74514999eeb034793c9"}
Oct 30 05:15:00.040147 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:00.039755    4874 scope.go:122] "RemoveContainer" containerID="238f5d8804dd5366de753024615f9b4783e452d8cbbf983397116663c3e742ce"
Oct 30 05:15:00.087297 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:00.087274    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx"]
Oct 30 05:15:00.091944 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:00.091924    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e070ea-f9c978495-ts7xx"]
Oct 30 05:15:00.728029 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:00.728000    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9be3939f-afda-4003-bd81-55e3349fdd86" path="/var/lib/kubelet/pods/9be3939f-afda-4003-bd81-55e3349fdd86/volumes"
Oct 30 05:15:00.913894 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:00.913859    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"]
Oct 30 05:15:00.914167 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:00.914111    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv" podUID="0d21e1c4-3882-438f-a391-b1697ff474cc" containerName="istio-proxy" containerID="cri-o://ee0042b90997842b7585fff3531ba7fb5d63126f318a8b8cb431b6c11c0610ae" gracePeriod=30
Oct 30 05:15:02.023192 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:02.023156    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s"]
Oct 30 05:15:02.023781 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:02.023425    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s" podUID="e05329a8-9320-47fe-9766-622ae3c93bc4" containerName="maas-api" containerID="cri-o://0a15d423ce486758eedac184592c3107ea361774d842826cc213f06acb78c9c2" gracePeriod=30
Oct 30 05:15:02.306628 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:02.306600    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s"
Oct 30 05:15:02.400312 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:02.400270    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e05329a8-9320-47fe-9766-622ae3c93bc4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e05329a8-9320-47fe-9766-622ae3c93bc4-openshift-service-ca\") pod \"e05329a8-9320-47fe-9766-622ae3c93bc4\" (UID: \"e05329a8-9320-47fe-9766-622ae3c93bc4\") "
Oct 30 05:15:02.400444 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:02.400334    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e05329a8-9320-47fe-9766-622ae3c93bc4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e05329a8-9320-47fe-9766-622ae3c93bc4-maas-api-tls\") pod \"e05329a8-9320-47fe-9766-622ae3c93bc4\" (UID: \"e05329a8-9320-47fe-9766-622ae3c93bc4\") "
Oct 30 05:15:02.400444 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:02.400409    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e05329a8-9320-47fe-9766-622ae3c93bc4-kube-api-access-r2vgv\" (UniqueName: \"kubernetes.io/projected/e05329a8-9320-47fe-9766-622ae3c93bc4-kube-api-access-r2vgv\") pod \"e05329a8-9320-47fe-9766-622ae3c93bc4\" (UID: \"e05329a8-9320-47fe-9766-622ae3c93bc4\") "
Oct 30 05:15:02.400544 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:02.400444    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e05329a8-9320-47fe-9766-622ae3c93bc4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e05329a8-9320-47fe-9766-622ae3c93bc4-metrics-certs\") pod \"e05329a8-9320-47fe-9766-622ae3c93bc4\" (UID: \"e05329a8-9320-47fe-9766-622ae3c93bc4\") "
Oct 30 05:15:02.400692 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:02.400662    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e05329a8-9320-47fe-9766-622ae3c93bc4-openshift-service-ca" pod "e05329a8-9320-47fe-9766-622ae3c93bc4" (UID: "e05329a8-9320-47fe-9766-622ae3c93bc4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:15:02.402551 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:02.402525    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e05329a8-9320-47fe-9766-622ae3c93bc4-maas-api-tls" pod "e05329a8-9320-47fe-9766-622ae3c93bc4" (UID: "e05329a8-9320-47fe-9766-622ae3c93bc4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:15:02.402656 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:02.402588    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e05329a8-9320-47fe-9766-622ae3c93bc4-metrics-certs" pod "e05329a8-9320-47fe-9766-622ae3c93bc4" (UID: "e05329a8-9320-47fe-9766-622ae3c93bc4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:15:02.402656 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:02.402632    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e05329a8-9320-47fe-9766-622ae3c93bc4-kube-api-access-r2vgv" pod "e05329a8-9320-47fe-9766-622ae3c93bc4" (UID: "e05329a8-9320-47fe-9766-622ae3c93bc4"). InnerVolumeSpecName "kube-api-access-r2vgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:02.501482 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:02.501447    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e05329a8-9320-47fe-9766-622ae3c93bc4-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:02.501482 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:02.501479    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e05329a8-9320-47fe-9766-622ae3c93bc4-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:02.501482 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:02.501489    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2vgv\" (UniqueName: \"kubernetes.io/projected/e05329a8-9320-47fe-9766-622ae3c93bc4-kube-api-access-r2vgv\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:02.501726 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:02.501498    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e05329a8-9320-47fe-9766-622ae3c93bc4-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:02.723919 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:02.723831    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj" podUID="6e900ea6-5bdb-40c7-bcd2-de0a6910aac0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:15:03.057745 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:03.057711    4874 generic.go:356] "Generic (PLEG): container finished" podID="e05329a8-9320-47fe-9766-622ae3c93bc4" containerID="0a15d423ce486758eedac184592c3107ea361774d842826cc213f06acb78c9c2" exitCode=0
Oct 30 05:15:03.058197 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:03.057760    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s"
Oct 30 05:15:03.058197 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:03.057804    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s" event={"ID":"e05329a8-9320-47fe-9766-622ae3c93bc4","Type":"ContainerDied","Data":"0a15d423ce486758eedac184592c3107ea361774d842826cc213f06acb78c9c2"}
Oct 30 05:15:03.058197 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:03.057854    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s" event={"ID":"e05329a8-9320-47fe-9766-622ae3c93bc4","Type":"ContainerDied","Data":"9f731bfba0cdebccd0b01b12ae64a5e6e8d54f1eada489196afc81c3c619800d"}
Oct 30 05:15:03.058197 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:03.057890    4874 scope.go:122] "RemoveContainer" containerID="0a15d423ce486758eedac184592c3107ea361774d842826cc213f06acb78c9c2"
Oct 30 05:15:03.066521 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:03.066504    4874 scope.go:122] "RemoveContainer" containerID="0a15d423ce486758eedac184592c3107ea361774d842826cc213f06acb78c9c2"
Oct 30 05:15:03.066777 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:15:03.066757    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a15d423ce486758eedac184592c3107ea361774d842826cc213f06acb78c9c2\": container with ID starting with 0a15d423ce486758eedac184592c3107ea361774d842826cc213f06acb78c9c2 not found: ID does not exist" containerID="0a15d423ce486758eedac184592c3107ea361774d842826cc213f06acb78c9c2"
Oct 30 05:15:03.066832 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:03.066784    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a15d423ce486758eedac184592c3107ea361774d842826cc213f06acb78c9c2"} err="failed to get container status \"0a15d423ce486758eedac184592c3107ea361774d842826cc213f06acb78c9c2\": rpc error: code = NotFound desc = could not find container \"0a15d423ce486758eedac184592c3107ea361774d842826cc213f06acb78c9c2\": container with ID starting with 0a15d423ce486758eedac184592c3107ea361774d842826cc213f06acb78c9c2 not found: ID does not exist"
Oct 30 05:15:03.106868 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:03.106844    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s"]
Oct 30 05:15:03.118117 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:03.118094    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-495ebe-8b4b586b4-fbr7s"]
Oct 30 05:15:04.727311 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:04.727277    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e05329a8-9320-47fe-9766-622ae3c93bc4" path="/var/lib/kubelet/pods/e05329a8-9320-47fe-9766-622ae3c93bc4/volumes"
Oct 30 05:15:04.792494 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:04.792460    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"]
Oct 30 05:15:04.792828 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:04.792762    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h" podUID="d5587cde-1301-458b-80c5-5a83682b3e21" containerName="istio-proxy" containerID="cri-o://29159fcdcf1a27941446d90960fd804b6bac1b4db837360519aa6865bdce46b2" gracePeriod=30
Oct 30 05:15:06.074259 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.074217    4874 generic.go:356] "Generic (PLEG): container finished" podID="0d21e1c4-3882-438f-a391-b1697ff474cc" containerID="ee0042b90997842b7585fff3531ba7fb5d63126f318a8b8cb431b6c11c0610ae" exitCode=0
Oct 30 05:15:06.074751 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.074294    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv" event={"ID":"0d21e1c4-3882-438f-a391-b1697ff474cc","Type":"ContainerDied","Data":"ee0042b90997842b7585fff3531ba7fb5d63126f318a8b8cb431b6c11c0610ae"}
Oct 30 05:15:06.162909 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.162885    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:15:06.235970 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.235943    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-podinfo\") pod \"0d21e1c4-3882-438f-a391-b1697ff474cc\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") "
Oct 30 05:15:06.236119 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.235993    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d21e1c4-3882-438f-a391-b1697ff474cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d21e1c4-3882-438f-a391-b1697ff474cc-istiod-ca-cert\") pod \"0d21e1c4-3882-438f-a391-b1697ff474cc\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") "
Oct 30 05:15:06.236119 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.236098    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-envoy\") pod \"0d21e1c4-3882-438f-a391-b1697ff474cc\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") "
Oct 30 05:15:06.236219 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.236139    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-token\" (UniqueName: \"kubernetes.io/projected/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-token\") pod \"0d21e1c4-3882-438f-a391-b1697ff474cc\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") "
Oct 30 05:15:06.236219 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.236160    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d21e1c4-3882-438f-a391-b1697ff474cc-kube-api-access-gtwmz\" (UniqueName: \"kubernetes.io/projected/0d21e1c4-3882-438f-a391-b1697ff474cc-kube-api-access-gtwmz\") pod \"0d21e1c4-3882-438f-a391-b1697ff474cc\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") "
Oct 30 05:15:06.236219 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.236201    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-credential-socket\") pod \"0d21e1c4-3882-438f-a391-b1697ff474cc\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") "
Oct 30 05:15:06.236420 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.236318    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-workload-socket\") pod \"0d21e1c4-3882-438f-a391-b1697ff474cc\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") "
Oct 30 05:15:06.236420 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.236379    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-data\") pod \"0d21e1c4-3882-438f-a391-b1697ff474cc\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") "
Oct 30 05:15:06.236420 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.236410    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-workload-certs\") pod \"0d21e1c4-3882-438f-a391-b1697ff474cc\" (UID: \"0d21e1c4-3882-438f-a391-b1697ff474cc\") "
Oct 30 05:15:06.236591 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.236445    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d21e1c4-3882-438f-a391-b1697ff474cc-istiod-ca-cert" pod "0d21e1c4-3882-438f-a391-b1697ff474cc" (UID: "0d21e1c4-3882-438f-a391-b1697ff474cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:15:06.236656 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.236597    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-workload-socket" pod "0d21e1c4-3882-438f-a391-b1697ff474cc" (UID: "0d21e1c4-3882-438f-a391-b1697ff474cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:06.236911 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.236721    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-data" pod "0d21e1c4-3882-438f-a391-b1697ff474cc" (UID: "0d21e1c4-3882-438f-a391-b1697ff474cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:06.236911 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.236819    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-workload-certs" pod "0d21e1c4-3882-438f-a391-b1697ff474cc" (UID: "0d21e1c4-3882-438f-a391-b1697ff474cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:06.236911 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.236840    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-credential-socket" pod "0d21e1c4-3882-438f-a391-b1697ff474cc" (UID: "0d21e1c4-3882-438f-a391-b1697ff474cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:06.236911 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.236857    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d21e1c4-3882-438f-a391-b1697ff474cc-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:06.236911 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.236876    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:06.236911 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.236888    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:06.238397 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.238372    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-envoy" pod "0d21e1c4-3882-438f-a391-b1697ff474cc" (UID: "0d21e1c4-3882-438f-a391-b1697ff474cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:06.238397 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.238388    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-token" pod "0d21e1c4-3882-438f-a391-b1697ff474cc" (UID: "0d21e1c4-3882-438f-a391-b1697ff474cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:06.238570 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.238553    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-podinfo" pod "0d21e1c4-3882-438f-a391-b1697ff474cc" (UID: "0d21e1c4-3882-438f-a391-b1697ff474cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:15:06.238729 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.238712    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d21e1c4-3882-438f-a391-b1697ff474cc-kube-api-access-gtwmz" pod "0d21e1c4-3882-438f-a391-b1697ff474cc" (UID: "0d21e1c4-3882-438f-a391-b1697ff474cc"). InnerVolumeSpecName "kube-api-access-gtwmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:06.337395 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.337367    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:06.337395 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.337391    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:06.337551 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.337401    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:06.337551 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.337409    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:06.337551 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.337417    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d21e1c4-3882-438f-a391-b1697ff474cc-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:06.337551 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.337424    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtwmz\" (UniqueName: \"kubernetes.io/projected/0d21e1c4-3882-438f-a391-b1697ff474cc-kube-api-access-gtwmz\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:06.416776 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:06.416746    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"]
Oct 30 05:15:07.079723 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:07.079643    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"
Oct 30 05:15:07.080184 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:07.079646    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv" event={"ID":"0d21e1c4-3882-438f-a391-b1697ff474cc","Type":"ContainerDied","Data":"4e44586e9d6633d5adb9e95caff5e13997930586e808ef6a22e08bcb6fa6cb08"}
Oct 30 05:15:07.080184 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:07.079772    4874 scope.go:122] "RemoveContainer" containerID="ee0042b90997842b7585fff3531ba7fb5d63126f318a8b8cb431b6c11c0610ae"
Oct 30 05:15:07.128440 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:07.128411    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"]
Oct 30 05:15:07.139671 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:07.139647    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-4f6mv"]
Oct 30 05:15:08.727053 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:08.727009    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d21e1c4-3882-438f-a391-b1697ff474cc" path="/var/lib/kubelet/pods/0d21e1c4-3882-438f-a391-b1697ff474cc/volumes"
Oct 30 05:15:10.039385 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.039359    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 05:15:10.096782 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.096713    4874 generic.go:356] "Generic (PLEG): container finished" podID="d5587cde-1301-458b-80c5-5a83682b3e21" containerID="29159fcdcf1a27941446d90960fd804b6bac1b4db837360519aa6865bdce46b2" exitCode=0
Oct 30 05:15:10.096782 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.096759    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"
Oct 30 05:15:10.096962 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.096804    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h" event={"ID":"d5587cde-1301-458b-80c5-5a83682b3e21","Type":"ContainerDied","Data":"29159fcdcf1a27941446d90960fd804b6bac1b4db837360519aa6865bdce46b2"}
Oct 30 05:15:10.096962 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.096878    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h" event={"ID":"d5587cde-1301-458b-80c5-5a83682b3e21","Type":"ContainerDied","Data":"9a2d51d55435a2285a44e0ea3c4b88e8bc097dd48ede42ac4e7f32b549ddcfac"}
Oct 30 05:15:10.096962 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.096901    4874 scope.go:122] "RemoveContainer" containerID="29159fcdcf1a27941446d90960fd804b6bac1b4db837360519aa6865bdce46b2"
Oct 30 05:15:10.105674 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.105654    4874 scope.go:122] "RemoveContainer" containerID="94a89367cef3a0bb978ace6f7734f152e213169061d6bf73441c286f51180167"
Oct 30 05:15:10.113463 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.113399    4874 scope.go:122] "RemoveContainer" containerID="29159fcdcf1a27941446d90960fd804b6bac1b4db837360519aa6865bdce46b2"
Oct 30 05:15:10.113686 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:15:10.113666    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29159fcdcf1a27941446d90960fd804b6bac1b4db837360519aa6865bdce46b2\": container with ID starting with 29159fcdcf1a27941446d90960fd804b6bac1b4db837360519aa6865bdce46b2 not found: ID does not exist" containerID="29159fcdcf1a27941446d90960fd804b6bac1b4db837360519aa6865bdce46b2"
Oct 30 05:15:10.113769 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.113693    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29159fcdcf1a27941446d90960fd804b6bac1b4db837360519aa6865bdce46b2"} err="failed to get container status \"29159fcdcf1a27941446d90960fd804b6bac1b4db837360519aa6865bdce46b2\": rpc error: code = NotFound desc = could not find container \"29159fcdcf1a27941446d90960fd804b6bac1b4db837360519aa6865bdce46b2\": container with ID starting with 29159fcdcf1a27941446d90960fd804b6bac1b4db837360519aa6865bdce46b2 not found: ID does not exist"
Oct 30 05:15:10.113769 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.113707    4874 scope.go:122] "RemoveContainer" containerID="94a89367cef3a0bb978ace6f7734f152e213169061d6bf73441c286f51180167"
Oct 30 05:15:10.113939 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:15:10.113919    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a89367cef3a0bb978ace6f7734f152e213169061d6bf73441c286f51180167\": container with ID starting with 94a89367cef3a0bb978ace6f7734f152e213169061d6bf73441c286f51180167 not found: ID does not exist" containerID="94a89367cef3a0bb978ace6f7734f152e213169061d6bf73441c286f51180167"
Oct 30 05:15:10.114086 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.113945    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a89367cef3a0bb978ace6f7734f152e213169061d6bf73441c286f51180167"} err="failed to get container status \"94a89367cef3a0bb978ace6f7734f152e213169061d6bf73441c286f51180167\": rpc error: code = NotFound desc = could not find container \"94a89367cef3a0bb978ace6f7734f152e213169061d6bf73441c286f51180167\": container with ID starting with 94a89367cef3a0bb978ace6f7734f152e213169061d6bf73441c286f51180167 not found: ID does not exist"
Oct 30 05:15:10.177032 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.177011    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-workload-certs\") pod \"d5587cde-1301-458b-80c5-5a83682b3e21\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") "
Oct 30 05:15:10.177135 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.177046    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5587cde-1301-458b-80c5-5a83682b3e21-istio-token\" (UniqueName: \"kubernetes.io/projected/d5587cde-1301-458b-80c5-5a83682b3e21-istio-token\") pod \"d5587cde-1301-458b-80c5-5a83682b3e21\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") "
Oct 30 05:15:10.177135 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.177112    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-credential-socket\") pod \"d5587cde-1301-458b-80c5-5a83682b3e21\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") "
Oct 30 05:15:10.177214 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.177131    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-workload-socket\") pod \"d5587cde-1301-458b-80c5-5a83682b3e21\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") "
Oct 30 05:15:10.177214 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.177159    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5587cde-1301-458b-80c5-5a83682b3e21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5587cde-1301-458b-80c5-5a83682b3e21-istio-podinfo\") pod \"d5587cde-1301-458b-80c5-5a83682b3e21\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") "
Oct 30 05:15:10.177214 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.177183    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-istio-envoy\") pod \"d5587cde-1301-458b-80c5-5a83682b3e21\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") "
Oct 30 05:15:10.177403 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.177213    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5587cde-1301-458b-80c5-5a83682b3e21-kube-api-access-k589t\" (UniqueName: \"kubernetes.io/projected/d5587cde-1301-458b-80c5-5a83682b3e21-kube-api-access-k589t\") pod \"d5587cde-1301-458b-80c5-5a83682b3e21\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") "
Oct 30 05:15:10.177403 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.177266    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5587cde-1301-458b-80c5-5a83682b3e21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5587cde-1301-458b-80c5-5a83682b3e21-istiod-ca-cert\") pod \"d5587cde-1301-458b-80c5-5a83682b3e21\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") "
Oct 30 05:15:10.177403 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.177312    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-workload-certs" pod "d5587cde-1301-458b-80c5-5a83682b3e21" (UID: "d5587cde-1301-458b-80c5-5a83682b3e21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:10.177403 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.177351    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-istio-data\") pod \"d5587cde-1301-458b-80c5-5a83682b3e21\" (UID: \"d5587cde-1301-458b-80c5-5a83682b3e21\") "
Oct 30 05:15:10.177626 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.177468    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-credential-socket" pod "d5587cde-1301-458b-80c5-5a83682b3e21" (UID: "d5587cde-1301-458b-80c5-5a83682b3e21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:10.177626 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.177472    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-workload-socket" pod "d5587cde-1301-458b-80c5-5a83682b3e21" (UID: "d5587cde-1301-458b-80c5-5a83682b3e21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:10.177772 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.177739    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:10.177772 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.177759    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:10.177900 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.177774    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:10.177900 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.177778    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-istio-data" pod "d5587cde-1301-458b-80c5-5a83682b3e21" (UID: "d5587cde-1301-458b-80c5-5a83682b3e21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:10.177900 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.177870    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5587cde-1301-458b-80c5-5a83682b3e21-istiod-ca-cert" pod "d5587cde-1301-458b-80c5-5a83682b3e21" (UID: "d5587cde-1301-458b-80c5-5a83682b3e21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:15:10.179414 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.179385    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-istio-envoy" pod "d5587cde-1301-458b-80c5-5a83682b3e21" (UID: "d5587cde-1301-458b-80c5-5a83682b3e21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:10.179504 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.179454    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5587cde-1301-458b-80c5-5a83682b3e21-kube-api-access-k589t" pod "d5587cde-1301-458b-80c5-5a83682b3e21" (UID: "d5587cde-1301-458b-80c5-5a83682b3e21"). InnerVolumeSpecName "kube-api-access-k589t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:10.179504 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.179465    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5587cde-1301-458b-80c5-5a83682b3e21-istio-token" pod "d5587cde-1301-458b-80c5-5a83682b3e21" (UID: "d5587cde-1301-458b-80c5-5a83682b3e21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:10.179583 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.179499    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5587cde-1301-458b-80c5-5a83682b3e21-istio-podinfo" pod "d5587cde-1301-458b-80c5-5a83682b3e21" (UID: "d5587cde-1301-458b-80c5-5a83682b3e21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:15:10.278890 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.278865    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5587cde-1301-458b-80c5-5a83682b3e21-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:10.278890 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.278888    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5587cde-1301-458b-80c5-5a83682b3e21-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:10.279059 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.278898    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:10.279059 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.278907    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k589t\" (UniqueName: \"kubernetes.io/projected/d5587cde-1301-458b-80c5-5a83682b3e21-kube-api-access-k589t\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:10.279059 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.278917    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5587cde-1301-458b-80c5-5a83682b3e21-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:10.279059 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.278924    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5587cde-1301-458b-80c5-5a83682b3e21-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:10.440095 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.440062    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"]
Oct 30 05:15:10.451303 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.451274    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-q447h"]
Oct 30 05:15:10.727879 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:10.727846    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5587cde-1301-458b-80c5-5a83682b3e21" path="/var/lib/kubelet/pods/d5587cde-1301-458b-80c5-5a83682b3e21/volumes"
Oct 30 05:15:11.519271 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:11.519229    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj"
Oct 30 05:15:11.690356 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:11.690270    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e900ea6-5bdb-40c7-bcd2-de0a6910aac0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e900ea6-5bdb-40c7-bcd2-de0a6910aac0-plugins-config-volume\") pod \"6e900ea6-5bdb-40c7-bcd2-de0a6910aac0\" (UID: \"6e900ea6-5bdb-40c7-bcd2-de0a6910aac0\") "
Oct 30 05:15:11.690356 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:11.690324    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e900ea6-5bdb-40c7-bcd2-de0a6910aac0-kube-api-access-8j5g7\" (UniqueName: \"kubernetes.io/projected/6e900ea6-5bdb-40c7-bcd2-de0a6910aac0-kube-api-access-8j5g7\") pod \"6e900ea6-5bdb-40c7-bcd2-de0a6910aac0\" (UID: \"6e900ea6-5bdb-40c7-bcd2-de0a6910aac0\") "
Oct 30 05:15:11.690732 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:11.690707    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e900ea6-5bdb-40c7-bcd2-de0a6910aac0-plugins-config-volume" pod "6e900ea6-5bdb-40c7-bcd2-de0a6910aac0" (UID: "6e900ea6-5bdb-40c7-bcd2-de0a6910aac0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:15:11.692474 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:11.692451    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e900ea6-5bdb-40c7-bcd2-de0a6910aac0-kube-api-access-8j5g7" pod "6e900ea6-5bdb-40c7-bcd2-de0a6910aac0" (UID: "6e900ea6-5bdb-40c7-bcd2-de0a6910aac0"). InnerVolumeSpecName "kube-api-access-8j5g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:11.791610 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:11.791572    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e900ea6-5bdb-40c7-bcd2-de0a6910aac0-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:11.791610 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:11.791602    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j5g7\" (UniqueName: \"kubernetes.io/projected/6e900ea6-5bdb-40c7-bcd2-de0a6910aac0-kube-api-access-8j5g7\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:12.108569 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:12.108531    4874 generic.go:356] "Generic (PLEG): container finished" podID="6e900ea6-5bdb-40c7-bcd2-de0a6910aac0" containerID="6a93dc9501e984d7b9dfbb1b22493e35ce6fb6da3432a1e1e502c0d12deab77e" exitCode=0
Oct 30 05:15:12.108833 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:12.108591    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj"
Oct 30 05:15:12.108962 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:12.108613    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj" event={"ID":"6e900ea6-5bdb-40c7-bcd2-de0a6910aac0","Type":"ContainerDied","Data":"6a93dc9501e984d7b9dfbb1b22493e35ce6fb6da3432a1e1e502c0d12deab77e"}
Oct 30 05:15:12.108962 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:12.108936    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj" event={"ID":"6e900ea6-5bdb-40c7-bcd2-de0a6910aac0","Type":"ContainerDied","Data":"ac8c8df7b4bf22fd8c53d070906151a7b792b2a1bef3822a985742556856db2c"}
Oct 30 05:15:12.108962 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:12.108957    4874 scope.go:122] "RemoveContainer" containerID="6a93dc9501e984d7b9dfbb1b22493e35ce6fb6da3432a1e1e502c0d12deab77e"
Oct 30 05:15:12.118164 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:12.118145    4874 scope.go:122] "RemoveContainer" containerID="6a93dc9501e984d7b9dfbb1b22493e35ce6fb6da3432a1e1e502c0d12deab77e"
Oct 30 05:15:12.118478 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:15:12.118456    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a93dc9501e984d7b9dfbb1b22493e35ce6fb6da3432a1e1e502c0d12deab77e\": container with ID starting with 6a93dc9501e984d7b9dfbb1b22493e35ce6fb6da3432a1e1e502c0d12deab77e not found: ID does not exist" containerID="6a93dc9501e984d7b9dfbb1b22493e35ce6fb6da3432a1e1e502c0d12deab77e"
Oct 30 05:15:12.118551 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:12.118485    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a93dc9501e984d7b9dfbb1b22493e35ce6fb6da3432a1e1e502c0d12deab77e"} err="failed to get container status \"6a93dc9501e984d7b9dfbb1b22493e35ce6fb6da3432a1e1e502c0d12deab77e\": rpc error: code = NotFound desc = could not find container \"6a93dc9501e984d7b9dfbb1b22493e35ce6fb6da3432a1e1e502c0d12deab77e\": container with ID starting with 6a93dc9501e984d7b9dfbb1b22493e35ce6fb6da3432a1e1e502c0d12deab77e not found: ID does not exist"
Oct 30 05:15:12.156544 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:12.156517    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj"]
Oct 30 05:15:12.167694 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:12.167670    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-247158-7c8957d7bd-nvvnj"]
Oct 30 05:15:12.727427 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:12.727386    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e900ea6-5bdb-40c7-bcd2-de0a6910aac0" path="/var/lib/kubelet/pods/6e900ea6-5bdb-40c7-bcd2-de0a6910aac0/volumes"
Oct 30 05:15:13.043730 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:13.043657    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:15:17.494385 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:17.494351    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj"]
Oct 30 05:15:21.436118 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:21.436074    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9" podUID="734ec4d9-3542-4a3b-ba5b-d47f3ddf048e" containerName="main" containerID="cri-o://5b02fe924e4d1b2bc384abbca1922484fd3c8c496723ad598292d2503880cfbd" gracePeriod=45
Oct 30 05:15:21.700092 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:21.700022    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 05:15:21.785088 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:21.785050    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-tls-certs\" (UniqueName: \"kubernetes.io/secret/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-tls-certs\") pod \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\" (UID: \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\") "
Oct 30 05:15:21.785311 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:21.785132    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-kube-api-access-7v9hc\" (UniqueName: \"kubernetes.io/projected/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-kube-api-access-7v9hc\") pod \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\" (UID: \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\") "
Oct 30 05:15:21.785311 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:21.785207    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-tmp-dir\") pod \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\" (UID: \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\") "
Oct 30 05:15:21.785311 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:21.785253    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-home\" (UniqueName: \"kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-home\") pod \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\" (UID: \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\") "
Oct 30 05:15:21.785311 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:21.785292    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-dshm\") pod \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\" (UID: \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\") "
Oct 30 05:15:21.785562 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:21.785330    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-model-cache\") pod \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\" (UID: \"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e\") "
Oct 30 05:15:21.785655 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:21.785626    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-home" pod "734ec4d9-3542-4a3b-ba5b-d47f3ddf048e" (UID: "734ec4d9-3542-4a3b-ba5b-d47f3ddf048e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:21.785787 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:21.785748    4874 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-home\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:21.785787 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:21.785755    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-tmp-dir" pod "734ec4d9-3542-4a3b-ba5b-d47f3ddf048e" (UID: "734ec4d9-3542-4a3b-ba5b-d47f3ddf048e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:21.786020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:21.785892    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-model-cache" pod "734ec4d9-3542-4a3b-ba5b-d47f3ddf048e" (UID: "734ec4d9-3542-4a3b-ba5b-d47f3ddf048e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:21.787625 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:21.787553    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-kube-api-access-7v9hc" pod "734ec4d9-3542-4a3b-ba5b-d47f3ddf048e" (UID: "734ec4d9-3542-4a3b-ba5b-d47f3ddf048e"). InnerVolumeSpecName "kube-api-access-7v9hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:21.787793 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:21.787768    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-tls-certs" pod "734ec4d9-3542-4a3b-ba5b-d47f3ddf048e" (UID: "734ec4d9-3542-4a3b-ba5b-d47f3ddf048e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:15:21.788967 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:21.788942    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-dshm" pod "734ec4d9-3542-4a3b-ba5b-d47f3ddf048e" (UID: "734ec4d9-3542-4a3b-ba5b-d47f3ddf048e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:15:21.886400 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:21.886357    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v9hc\" (UniqueName: \"kubernetes.io/projected/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-kube-api-access-7v9hc\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:21.886400 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:21.886399    4874 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-tmp-dir\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:21.886599 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:21.886418    4874 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-dshm\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:21.886599 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:21.886431    4874 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-model-cache\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:21.886599 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:21.886444    4874 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e-tls-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:22.164878 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:22.164842    4874 generic.go:356] "Generic (PLEG): container finished" podID="734ec4d9-3542-4a3b-ba5b-d47f3ddf048e" containerID="5b02fe924e4d1b2bc384abbca1922484fd3c8c496723ad598292d2503880cfbd" exitCode=0
Oct 30 05:15:22.164878 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:22.164887    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"
Oct 30 05:15:22.165164 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:22.164932    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9" event={"ID":"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e","Type":"ContainerDied","Data":"5b02fe924e4d1b2bc384abbca1922484fd3c8c496723ad598292d2503880cfbd"}
Oct 30 05:15:22.165164 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:22.164984    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9" event={"ID":"734ec4d9-3542-4a3b-ba5b-d47f3ddf048e","Type":"ContainerDied","Data":"4645706347eec0c6b72aec939364d0026be939ca3d1f6ffd901caef1e73f502f"}
Oct 30 05:15:22.165164 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:22.165008    4874 scope.go:122] "RemoveContainer" containerID="5b02fe924e4d1b2bc384abbca1922484fd3c8c496723ad598292d2503880cfbd"
Oct 30 05:15:22.175360 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:22.175333    4874 scope.go:122] "RemoveContainer" containerID="5b02fe924e4d1b2bc384abbca1922484fd3c8c496723ad598292d2503880cfbd"
Oct 30 05:15:22.175689 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:15:22.175661    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b02fe924e4d1b2bc384abbca1922484fd3c8c496723ad598292d2503880cfbd\": container with ID starting with 5b02fe924e4d1b2bc384abbca1922484fd3c8c496723ad598292d2503880cfbd not found: ID does not exist" containerID="5b02fe924e4d1b2bc384abbca1922484fd3c8c496723ad598292d2503880cfbd"
Oct 30 05:15:22.175783 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:22.175701    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b02fe924e4d1b2bc384abbca1922484fd3c8c496723ad598292d2503880cfbd"} err="failed to get container status \"5b02fe924e4d1b2bc384abbca1922484fd3c8c496723ad598292d2503880cfbd\": rpc error: code = NotFound desc = could not find container \"5b02fe924e4d1b2bc384abbca1922484fd3c8c496723ad598292d2503880cfbd\": container with ID starting with 5b02fe924e4d1b2bc384abbca1922484fd3c8c496723ad598292d2503880cfbd not found: ID does not exist"
Oct 30 05:15:22.212130 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:22.212102    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"]
Oct 30 05:15:22.215833 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:22.215806    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-36bbac/premium-w0-36bbac-kserve-6ccf4bfbfc-kd4n9"]
Oct 30 05:15:22.507781 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:22.506331    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj" podUID="a54bdfac-68a5-4f09-bc41-fd51c3209245" containerName="payload-processing" containerID="cri-o://b96590df5c160a030e0e964170d48f48c760a99a795200d0b79f114009dc7e7c" gracePeriod=25
Oct 30 05:15:22.728809 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:22.728773    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="734ec4d9-3542-4a3b-ba5b-d47f3ddf048e" path="/var/lib/kubelet/pods/734ec4d9-3542-4a3b-ba5b-d47f3ddf048e/volumes"
Oct 30 05:15:25.611954 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:15:25.611906    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:15:28.394977 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:28.394945    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd"]
Oct 30 05:15:28.395589 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:28.395140    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd" podUID="96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08" containerName="maas-api" containerID="cri-o://e1c24733c803a9b80e10ad3c4166a08a0099863d7503ea3ceeacea09cf87db3d" gracePeriod=30
Oct 30 05:15:28.648265 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:28.648170    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd"
Oct 30 05:15:28.671355 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:28.671325    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-openshift-service-ca\") pod \"96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08\" (UID: \"96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08\") "
Oct 30 05:15:28.671558 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:28.671375    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-kube-api-access-vw924\" (UniqueName: \"kubernetes.io/projected/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-kube-api-access-vw924\") pod \"96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08\" (UID: \"96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08\") "
Oct 30 05:15:28.671558 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:28.671409    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-metrics-certs\" (UniqueName: \"kubernetes.io/secret/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-metrics-certs\") pod \"96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08\" (UID: \"96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08\") "
Oct 30 05:15:28.671558 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:28.671435    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-maas-api-tls\") pod \"96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08\" (UID: \"96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08\") "
Oct 30 05:15:28.671758 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:28.671694    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-openshift-service-ca" pod "96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08" (UID: "96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:15:28.673631 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:28.673598    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-metrics-certs" pod "96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08" (UID: "96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:15:28.673631 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:28.673621    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-maas-api-tls" pod "96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08" (UID: "96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:15:28.673851 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:28.673681    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-kube-api-access-vw924" pod "96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08" (UID: "96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08"). InnerVolumeSpecName "kube-api-access-vw924". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:28.762698 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:28.762668    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268"]
Oct 30 05:15:28.762863 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:28.762842    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268" podUID="245deabe-bbd0-42ef-b7dd-c264c38acdb8" containerName="maas-api" containerID="cri-o://04abf9fd437529d169294f5bdedc7966d9fe7455729e77601810322692586378" gracePeriod=30
Oct 30 05:15:28.773505 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:28.773069    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:28.773505 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:28.773102    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw924\" (UniqueName: \"kubernetes.io/projected/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-kube-api-access-vw924\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:28.773505 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:28.773117    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:28.773505 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:28.773130    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:29.032455 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.032411    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268"
Oct 30 05:15:29.076276 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.076232    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/245deabe-bbd0-42ef-b7dd-c264c38acdb8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/245deabe-bbd0-42ef-b7dd-c264c38acdb8-metrics-certs\") pod \"245deabe-bbd0-42ef-b7dd-c264c38acdb8\" (UID: \"245deabe-bbd0-42ef-b7dd-c264c38acdb8\") "
Oct 30 05:15:29.076484 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.076313    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/245deabe-bbd0-42ef-b7dd-c264c38acdb8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/245deabe-bbd0-42ef-b7dd-c264c38acdb8-openshift-service-ca\") pod \"245deabe-bbd0-42ef-b7dd-c264c38acdb8\" (UID: \"245deabe-bbd0-42ef-b7dd-c264c38acdb8\") "
Oct 30 05:15:29.076484 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.076356    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/245deabe-bbd0-42ef-b7dd-c264c38acdb8-kube-api-access-k87lc\" (UniqueName: \"kubernetes.io/projected/245deabe-bbd0-42ef-b7dd-c264c38acdb8-kube-api-access-k87lc\") pod \"245deabe-bbd0-42ef-b7dd-c264c38acdb8\" (UID: \"245deabe-bbd0-42ef-b7dd-c264c38acdb8\") "
Oct 30 05:15:29.076484 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.076384    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/245deabe-bbd0-42ef-b7dd-c264c38acdb8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/245deabe-bbd0-42ef-b7dd-c264c38acdb8-maas-api-tls\") pod \"245deabe-bbd0-42ef-b7dd-c264c38acdb8\" (UID: \"245deabe-bbd0-42ef-b7dd-c264c38acdb8\") "
Oct 30 05:15:29.076707 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.076682    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/245deabe-bbd0-42ef-b7dd-c264c38acdb8-openshift-service-ca" pod "245deabe-bbd0-42ef-b7dd-c264c38acdb8" (UID: "245deabe-bbd0-42ef-b7dd-c264c38acdb8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:15:29.078623 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.078574    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/245deabe-bbd0-42ef-b7dd-c264c38acdb8-metrics-certs" pod "245deabe-bbd0-42ef-b7dd-c264c38acdb8" (UID: "245deabe-bbd0-42ef-b7dd-c264c38acdb8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:15:29.078623 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.078591    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/245deabe-bbd0-42ef-b7dd-c264c38acdb8-maas-api-tls" pod "245deabe-bbd0-42ef-b7dd-c264c38acdb8" (UID: "245deabe-bbd0-42ef-b7dd-c264c38acdb8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:15:29.078823 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.078674    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245deabe-bbd0-42ef-b7dd-c264c38acdb8-kube-api-access-k87lc" pod "245deabe-bbd0-42ef-b7dd-c264c38acdb8" (UID: "245deabe-bbd0-42ef-b7dd-c264c38acdb8"). InnerVolumeSpecName "kube-api-access-k87lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:29.177517 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.177473    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/245deabe-bbd0-42ef-b7dd-c264c38acdb8-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:29.177517 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.177513    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/245deabe-bbd0-42ef-b7dd-c264c38acdb8-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:29.177517 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.177523    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k87lc\" (UniqueName: \"kubernetes.io/projected/245deabe-bbd0-42ef-b7dd-c264c38acdb8-kube-api-access-k87lc\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:29.177517 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.177534    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/245deabe-bbd0-42ef-b7dd-c264c38acdb8-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:29.202127 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.202040    4874 generic.go:356] "Generic (PLEG): container finished" podID="245deabe-bbd0-42ef-b7dd-c264c38acdb8" containerID="04abf9fd437529d169294f5bdedc7966d9fe7455729e77601810322692586378" exitCode=0
Oct 30 05:15:29.202127 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.202088    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268"
Oct 30 05:15:29.202409 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.202082    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268" event={"ID":"245deabe-bbd0-42ef-b7dd-c264c38acdb8","Type":"ContainerDied","Data":"04abf9fd437529d169294f5bdedc7966d9fe7455729e77601810322692586378"}
Oct 30 05:15:29.202409 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.202215    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268" event={"ID":"245deabe-bbd0-42ef-b7dd-c264c38acdb8","Type":"ContainerDied","Data":"ba85d12cffa40005dea1c2a40b7ad266e1e4768de63affce8a694a9ba0f7b908"}
Oct 30 05:15:29.202409 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.202268    4874 scope.go:122] "RemoveContainer" containerID="04abf9fd437529d169294f5bdedc7966d9fe7455729e77601810322692586378"
Oct 30 05:15:29.213589 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.212012    4874 generic.go:356] "Generic (PLEG): container finished" podID="96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08" containerID="e1c24733c803a9b80e10ad3c4166a08a0099863d7503ea3ceeacea09cf87db3d" exitCode=0
Oct 30 05:15:29.213589 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.212181    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd" event={"ID":"96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08","Type":"ContainerDied","Data":"e1c24733c803a9b80e10ad3c4166a08a0099863d7503ea3ceeacea09cf87db3d"}
Oct 30 05:15:29.213589 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.212307    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd" event={"ID":"96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08","Type":"ContainerDied","Data":"839ab1e346a0a66b83b433dd80d3d53e85a6f3747174a95b35549d0245413558"}
Oct 30 05:15:29.213589 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.212234    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd"
Oct 30 05:15:29.223642 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.223623    4874 scope.go:122] "RemoveContainer" containerID="04abf9fd437529d169294f5bdedc7966d9fe7455729e77601810322692586378"
Oct 30 05:15:29.223945 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:15:29.223926    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04abf9fd437529d169294f5bdedc7966d9fe7455729e77601810322692586378\": container with ID starting with 04abf9fd437529d169294f5bdedc7966d9fe7455729e77601810322692586378 not found: ID does not exist" containerID="04abf9fd437529d169294f5bdedc7966d9fe7455729e77601810322692586378"
Oct 30 05:15:29.223993 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.223955    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04abf9fd437529d169294f5bdedc7966d9fe7455729e77601810322692586378"} err="failed to get container status \"04abf9fd437529d169294f5bdedc7966d9fe7455729e77601810322692586378\": rpc error: code = NotFound desc = could not find container \"04abf9fd437529d169294f5bdedc7966d9fe7455729e77601810322692586378\": container with ID starting with 04abf9fd437529d169294f5bdedc7966d9fe7455729e77601810322692586378 not found: ID does not exist"
Oct 30 05:15:29.223993 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.223974    4874 scope.go:122] "RemoveContainer" containerID="e1c24733c803a9b80e10ad3c4166a08a0099863d7503ea3ceeacea09cf87db3d"
Oct 30 05:15:29.231656 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.231636    4874 scope.go:122] "RemoveContainer" containerID="e1c24733c803a9b80e10ad3c4166a08a0099863d7503ea3ceeacea09cf87db3d"
Oct 30 05:15:29.231899 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:15:29.231873    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1c24733c803a9b80e10ad3c4166a08a0099863d7503ea3ceeacea09cf87db3d\": container with ID starting with e1c24733c803a9b80e10ad3c4166a08a0099863d7503ea3ceeacea09cf87db3d not found: ID does not exist" containerID="e1c24733c803a9b80e10ad3c4166a08a0099863d7503ea3ceeacea09cf87db3d"
Oct 30 05:15:29.231995 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.231904    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c24733c803a9b80e10ad3c4166a08a0099863d7503ea3ceeacea09cf87db3d"} err="failed to get container status \"e1c24733c803a9b80e10ad3c4166a08a0099863d7503ea3ceeacea09cf87db3d\": rpc error: code = NotFound desc = could not find container \"e1c24733c803a9b80e10ad3c4166a08a0099863d7503ea3ceeacea09cf87db3d\": container with ID starting with e1c24733c803a9b80e10ad3c4166a08a0099863d7503ea3ceeacea09cf87db3d not found: ID does not exist"
Oct 30 05:15:29.266960 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.266937    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd"]
Oct 30 05:15:29.271002 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.270983    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-36bbac-9bd67cf86-nrsnd"]
Oct 30 05:15:29.303306 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.303281    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268"]
Oct 30 05:15:29.310958 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:29.310935    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b4708a-8bfc5c8db-st268"]
Oct 30 05:15:30.340602 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:30.340570    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:15:30.731584 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:30.731546    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="245deabe-bbd0-42ef-b7dd-c264c38acdb8" path="/var/lib/kubelet/pods/245deabe-bbd0-42ef-b7dd-c264c38acdb8/volumes"
Oct 30 05:15:30.731983 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:30.731967    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08" path="/var/lib/kubelet/pods/96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08/volumes"
Oct 30 05:15:31.542549 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:31.542516    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj" podUID="a54bdfac-68a5-4f09-bc41-fd51c3209245" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:15:35.096325 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:35.096293    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-757f4bf64c-szwjz_36ea66f0-e11f-4da2-a421-1498c0287776/authorino/0.log"
Oct 30 05:15:35.703039 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:35.703012    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vmffb_93fe6d14-eca6-492a-9237-75362698993f/kuadrant-console-plugin/0.log"
Oct 30 05:15:36.202339 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:36.202308    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-vg5td_b90621d3-240a-489c-9ed2-6c1913a13972/limitador/0.log"
Oct 30 05:15:41.542508 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:41.542468    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj" podUID="a54bdfac-68a5-4f09-bc41-fd51c3209245" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:15:42.765682 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:42.765659    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj"
Oct 30 05:15:42.898326 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:42.898226    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a54bdfac-68a5-4f09-bc41-fd51c3209245-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a54bdfac-68a5-4f09-bc41-fd51c3209245-plugins-config-volume\") pod \"a54bdfac-68a5-4f09-bc41-fd51c3209245\" (UID: \"a54bdfac-68a5-4f09-bc41-fd51c3209245\") "
Oct 30 05:15:42.898461 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:42.898376    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a54bdfac-68a5-4f09-bc41-fd51c3209245-kube-api-access-9bsf6\" (UniqueName: \"kubernetes.io/projected/a54bdfac-68a5-4f09-bc41-fd51c3209245-kube-api-access-9bsf6\") pod \"a54bdfac-68a5-4f09-bc41-fd51c3209245\" (UID: \"a54bdfac-68a5-4f09-bc41-fd51c3209245\") "
Oct 30 05:15:42.898585 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:42.898564    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54bdfac-68a5-4f09-bc41-fd51c3209245-plugins-config-volume" pod "a54bdfac-68a5-4f09-bc41-fd51c3209245" (UID: "a54bdfac-68a5-4f09-bc41-fd51c3209245"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:15:42.900418 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:42.900392    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54bdfac-68a5-4f09-bc41-fd51c3209245-kube-api-access-9bsf6" pod "a54bdfac-68a5-4f09-bc41-fd51c3209245" (UID: "a54bdfac-68a5-4f09-bc41-fd51c3209245"). InnerVolumeSpecName "kube-api-access-9bsf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:15:42.999511 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:42.999485    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bsf6\" (UniqueName: \"kubernetes.io/projected/a54bdfac-68a5-4f09-bc41-fd51c3209245-kube-api-access-9bsf6\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:42.999511 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:42.999510    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a54bdfac-68a5-4f09-bc41-fd51c3209245-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:15:43.097758 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:43.097723    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:15:43.276462 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:43.276427    4874 generic.go:356] "Generic (PLEG): container finished" podID="a54bdfac-68a5-4f09-bc41-fd51c3209245" containerID="b96590df5c160a030e0e964170d48f48c760a99a795200d0b79f114009dc7e7c" exitCode=0
Oct 30 05:15:43.276650 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:43.276473    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj"
Oct 30 05:15:43.276650 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:43.276508    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj" event={"ID":"a54bdfac-68a5-4f09-bc41-fd51c3209245","Type":"ContainerDied","Data":"b96590df5c160a030e0e964170d48f48c760a99a795200d0b79f114009dc7e7c"}
Oct 30 05:15:43.276650 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:43.276561    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj" event={"ID":"a54bdfac-68a5-4f09-bc41-fd51c3209245","Type":"ContainerDied","Data":"fd5d5302a407f7cfe093993dd91ce9a27159933e6d41718e12088616606bc140"}
Oct 30 05:15:43.276650 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:43.276583    4874 scope.go:122] "RemoveContainer" containerID="b96590df5c160a030e0e964170d48f48c760a99a795200d0b79f114009dc7e7c"
Oct 30 05:15:43.284812 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:43.284796    4874 scope.go:122] "RemoveContainer" containerID="b96590df5c160a030e0e964170d48f48c760a99a795200d0b79f114009dc7e7c"
Oct 30 05:15:43.285080 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:15:43.285061    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b96590df5c160a030e0e964170d48f48c760a99a795200d0b79f114009dc7e7c\": container with ID starting with b96590df5c160a030e0e964170d48f48c760a99a795200d0b79f114009dc7e7c not found: ID does not exist" containerID="b96590df5c160a030e0e964170d48f48c760a99a795200d0b79f114009dc7e7c"
Oct 30 05:15:43.285129 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:43.285088    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b96590df5c160a030e0e964170d48f48c760a99a795200d0b79f114009dc7e7c"} err="failed to get container status \"b96590df5c160a030e0e964170d48f48c760a99a795200d0b79f114009dc7e7c\": rpc error: code = NotFound desc = could not find container \"b96590df5c160a030e0e964170d48f48c760a99a795200d0b79f114009dc7e7c\": container with ID starting with b96590df5c160a030e0e964170d48f48c760a99a795200d0b79f114009dc7e7c not found: ID does not exist"
Oct 30 05:15:43.325184 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:43.325151    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj"]
Oct 30 05:15:43.337682 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:43.337656    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b4708a-fbc8b564-qq4sj"]
Oct 30 05:15:44.726735 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:44.726701    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a54bdfac-68a5-4f09-bc41-fd51c3209245" path="/var/lib/kubelet/pods/a54bdfac-68a5-4f09-bc41-fd51c3209245/volumes"
Oct 30 05:15:48.838326 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838289    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb"]
Oct 30 05:15:48.838856 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838745    4874 cpu_manager.go:414] "containerMap: removing container" podUID="9be3939f-afda-4003-bd81-55e3349fdd86" containerName="payload-pre-processing"
Oct 30 05:15:48.838856 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838763    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9be3939f-afda-4003-bd81-55e3349fdd86" containerName="payload-pre-processing"
Oct 30 05:15:48.838856 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838780    4874 cpu_manager.go:414] "containerMap: removing container" podUID="2ac6eaed-819d-4ce2-b299-b7228312da81" containerName="payload-pre-processing"
Oct 30 05:15:48.838856 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838787    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ac6eaed-819d-4ce2-b299-b7228312da81" containerName="payload-pre-processing"
Oct 30 05:15:48.838856 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838799    4874 cpu_manager.go:414] "containerMap: removing container" podUID="b9c8bf07-6e78-4a49-a32c-302fe418d7cc" containerName="payload-processing"
Oct 30 05:15:48.838856 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838805    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9c8bf07-6e78-4a49-a32c-302fe418d7cc" containerName="payload-processing"
Oct 30 05:15:48.838856 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838815    4874 cpu_manager.go:414] "containerMap: removing container" podUID="255ff92e-6bed-464a-a995-04cbc315cf99" containerName="maas-api"
Oct 30 05:15:48.838856 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838821    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="255ff92e-6bed-464a-a995-04cbc315cf99" containerName="maas-api"
Oct 30 05:15:48.838856 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838834    4874 cpu_manager.go:414] "containerMap: removing container" podUID="734ec4d9-3542-4a3b-ba5b-d47f3ddf048e" containerName="main"
Oct 30 05:15:48.838856 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838841    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="734ec4d9-3542-4a3b-ba5b-d47f3ddf048e" containerName="main"
Oct 30 05:15:48.838856 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838854    4874 cpu_manager.go:414] "containerMap: removing container" podUID="833c472c-0226-4e47-bc11-5d2659845a86" containerName="istio-proxy"
Oct 30 05:15:48.838856 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838862    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="833c472c-0226-4e47-bc11-5d2659845a86" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838878    4874 cpu_manager.go:414] "containerMap: removing container" podUID="d363d951-949c-4438-8c54-e65908025cfd" containerName="main"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838887    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d363d951-949c-4438-8c54-e65908025cfd" containerName="main"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838900    4874 cpu_manager.go:414] "containerMap: removing container" podUID="f3db5246-707f-4592-8ea0-f09ede62b5af" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838907    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3db5246-707f-4592-8ea0-f09ede62b5af" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838917    4874 cpu_manager.go:414] "containerMap: removing container" podUID="d605506d-43ec-447c-b573-cb4512867fec" containerName="main"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838924    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d605506d-43ec-447c-b573-cb4512867fec" containerName="main"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838935    4874 cpu_manager.go:414] "containerMap: removing container" podUID="7479a6e1-f414-411e-b1a0-a56fe33f8262" containerName="maas-api"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838941    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7479a6e1-f414-411e-b1a0-a56fe33f8262" containerName="maas-api"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838958    4874 cpu_manager.go:414] "containerMap: removing container" podUID="0e28b1d5-cbaa-40ea-b74c-fecb28d84a38" containerName="main"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838966    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e28b1d5-cbaa-40ea-b74c-fecb28d84a38" containerName="main"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838975    4874 cpu_manager.go:414] "containerMap: removing container" podUID="9b7e4826-cc1f-401c-ad81-608f4234cf7f" containerName="maas-api"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838982    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b7e4826-cc1f-401c-ad81-608f4234cf7f" containerName="maas-api"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.838995    4874 cpu_manager.go:414] "containerMap: removing container" podUID="14f989b1-ee76-4cef-950c-ed97b8a5b19b" containerName="maas-api"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839001    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14f989b1-ee76-4cef-950c-ed97b8a5b19b" containerName="maas-api"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839010    4874 cpu_manager.go:414] "containerMap: removing container" podUID="a849651f-2575-40e3-9f53-3c2b057075ce" containerName="maas-api"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839018    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a849651f-2575-40e3-9f53-3c2b057075ce" containerName="maas-api"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839029    4874 cpu_manager.go:414] "containerMap: removing container" podUID="c7abcd90-6f30-44af-bdc4-760820351783" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839036    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7abcd90-6f30-44af-bdc4-760820351783" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839046    4874 cpu_manager.go:414] "containerMap: removing container" podUID="c7abcd90-6f30-44af-bdc4-760820351783" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839053    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7abcd90-6f30-44af-bdc4-760820351783" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839062    4874 cpu_manager.go:414] "containerMap: removing container" podUID="b71a43c4-1c1e-4e92-9a3d-68b6827efa7b" containerName="payload-pre-processing"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839069    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b71a43c4-1c1e-4e92-9a3d-68b6827efa7b" containerName="payload-pre-processing"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839084    4874 cpu_manager.go:414] "containerMap: removing container" podUID="c4ec0ec0-354e-41f9-897e-ecbd08404b2e" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839091    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ec0ec0-354e-41f9-897e-ecbd08404b2e" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839102    4874 cpu_manager.go:414] "containerMap: removing container" podUID="1463c013-e615-4142-b799-56b17e67a474" containerName="main"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839109    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1463c013-e615-4142-b799-56b17e67a474" containerName="main"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839119    4874 cpu_manager.go:414] "containerMap: removing container" podUID="5b386201-0f5e-4add-98a8-bb3bcf476089" containerName="payload-pre-processing"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839127    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b386201-0f5e-4add-98a8-bb3bcf476089" containerName="payload-pre-processing"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839137    4874 cpu_manager.go:414] "containerMap: removing container" podUID="0d21e1c4-3882-438f-a391-b1697ff474cc" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839146    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d21e1c4-3882-438f-a391-b1697ff474cc" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839158    4874 cpu_manager.go:414] "containerMap: removing container" podUID="fde6ef20-18b2-4615-952e-8de495f731fe" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839165    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde6ef20-18b2-4615-952e-8de495f731fe" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839175    4874 cpu_manager.go:414] "containerMap: removing container" podUID="833c472c-0226-4e47-bc11-5d2659845a86" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839182    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="833c472c-0226-4e47-bc11-5d2659845a86" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839192    4874 cpu_manager.go:414] "containerMap: removing container" podUID="142fff76-0542-40d8-bea6-f46a50366b1d" containerName="maas-api"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839199    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="142fff76-0542-40d8-bea6-f46a50366b1d" containerName="maas-api"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839214    4874 cpu_manager.go:414] "containerMap: removing container" podUID="17cf01ad-84ee-4f36-92f0-7cfff29f4215" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839222    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17cf01ad-84ee-4f36-92f0-7cfff29f4215" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839233    4874 cpu_manager.go:414] "containerMap: removing container" podUID="96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08" containerName="maas-api"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839261    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08" containerName="maas-api"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839271    4874 cpu_manager.go:414] "containerMap: removing container" podUID="c4ec0ec0-354e-41f9-897e-ecbd08404b2e" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839279    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ec0ec0-354e-41f9-897e-ecbd08404b2e" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839288    4874 cpu_manager.go:414] "containerMap: removing container" podUID="20246613-3c91-4fc6-9c96-0e0b81709958" containerName="maas-api"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839295    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20246613-3c91-4fc6-9c96-0e0b81709958" containerName="maas-api"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839308    4874 cpu_manager.go:414] "containerMap: removing container" podUID="6e900ea6-5bdb-40c7-bcd2-de0a6910aac0" containerName="payload-processing"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839316    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e900ea6-5bdb-40c7-bcd2-de0a6910aac0" containerName="payload-processing"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839326    4874 cpu_manager.go:414] "containerMap: removing container" podUID="e05329a8-9320-47fe-9766-622ae3c93bc4" containerName="maas-api"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839334    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e05329a8-9320-47fe-9766-622ae3c93bc4" containerName="maas-api"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839344    4874 cpu_manager.go:414] "containerMap: removing container" podUID="54b0d7fa-4b81-49a9-9819-2ee76bd5177e" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839351    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54b0d7fa-4b81-49a9-9819-2ee76bd5177e" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839361    4874 cpu_manager.go:414] "containerMap: removing container" podUID="d5587cde-1301-458b-80c5-5a83682b3e21" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839368    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5587cde-1301-458b-80c5-5a83682b3e21" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839377    4874 cpu_manager.go:414] "containerMap: removing container" podUID="fde6ef20-18b2-4615-952e-8de495f731fe" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839384    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde6ef20-18b2-4615-952e-8de495f731fe" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839398    4874 cpu_manager.go:414] "containerMap: removing container" podUID="70ea964b-efdb-4d99-b4f5-e3cde4331254" containerName="main"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839405    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70ea964b-efdb-4d99-b4f5-e3cde4331254" containerName="main"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839418    4874 cpu_manager.go:414] "containerMap: removing container" podUID="245deabe-bbd0-42ef-b7dd-c264c38acdb8" containerName="maas-api"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839425    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="245deabe-bbd0-42ef-b7dd-c264c38acdb8" containerName="maas-api"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839435    4874 cpu_manager.go:414] "containerMap: removing container" podUID="d5587cde-1301-458b-80c5-5a83682b3e21" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839441    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5587cde-1301-458b-80c5-5a83682b3e21" containerName="istio-proxy"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839455    4874 cpu_manager.go:414] "containerMap: removing container" podUID="ea64a87b-85b1-46fa-9c51-4e45543bf808" containerName="maas-api"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839483    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea64a87b-85b1-46fa-9c51-4e45543bf808" containerName="maas-api"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839496    4874 cpu_manager.go:414] "containerMap: removing container" podUID="e1774f86-5e25-4358-b250-d26c6ea6c601" containerName="payload-processing"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839505    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1774f86-5e25-4358-b250-d26c6ea6c601" containerName="payload-processing"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839516    4874 cpu_manager.go:414] "containerMap: removing container" podUID="a54bdfac-68a5-4f09-bc41-fd51c3209245" containerName="payload-processing"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839524    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a54bdfac-68a5-4f09-bc41-fd51c3209245" containerName="payload-processing"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839535    4874 cpu_manager.go:414] "containerMap: removing container" podUID="beb04a4a-dc5c-4583-a464-3c56374faa33" containerName="payload-pre-processing"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839542    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beb04a4a-dc5c-4583-a464-3c56374faa33" containerName="payload-pre-processing"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839552    4874 cpu_manager.go:414] "containerMap: removing container" podUID="47a05334-0a58-4dd9-a9e3-2194031456cd" containerName="main"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839559    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47a05334-0a58-4dd9-a9e3-2194031456cd" containerName="main"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839568    4874 cpu_manager.go:414] "containerMap: removing container" podUID="653f8461-4436-4506-bc3e-0271173d425e" containerName="payload-processing"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839575    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="653f8461-4436-4506-bc3e-0271173d425e" containerName="payload-processing"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839585    4874 cpu_manager.go:414] "containerMap: removing container" podUID="8d278356-d7d2-42cd-bba4-e5ef1867fa1c" containerName="maas-api"
Oct 30 05:15:48.839554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839592    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d278356-d7d2-42cd-bba4-e5ef1867fa1c" containerName="maas-api"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839602    4874 cpu_manager.go:414] "containerMap: removing container" podUID="17cf01ad-84ee-4f36-92f0-7cfff29f4215" containerName="istio-proxy"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839609    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17cf01ad-84ee-4f36-92f0-7cfff29f4215" containerName="istio-proxy"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839697    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3db5246-707f-4592-8ea0-f09ede62b5af" containerName="istio-proxy"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839710    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="d605506d-43ec-447c-b573-cb4512867fec" containerName="main"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839722    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="7479a6e1-f414-411e-b1a0-a56fe33f8262" containerName="maas-api"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839731    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e28b1d5-cbaa-40ea-b74c-fecb28d84a38" containerName="main"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839739    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b7e4826-cc1f-401c-ad81-608f4234cf7f" containerName="maas-api"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839750    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="14f989b1-ee76-4cef-950c-ed97b8a5b19b" containerName="maas-api"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839759    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="a849651f-2575-40e3-9f53-3c2b057075ce" containerName="maas-api"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839767    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7abcd90-6f30-44af-bdc4-760820351783" containerName="istio-proxy"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839777    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7abcd90-6f30-44af-bdc4-760820351783" containerName="istio-proxy"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839785    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="b71a43c4-1c1e-4e92-9a3d-68b6827efa7b" containerName="payload-pre-processing"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839795    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4ec0ec0-354e-41f9-897e-ecbd08404b2e" containerName="istio-proxy"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839803    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="1463c013-e615-4142-b799-56b17e67a474" containerName="main"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839812    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b386201-0f5e-4add-98a8-bb3bcf476089" containerName="payload-pre-processing"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839820    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d21e1c4-3882-438f-a391-b1697ff474cc" containerName="istio-proxy"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839830    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde6ef20-18b2-4615-952e-8de495f731fe" containerName="istio-proxy"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839838    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="833c472c-0226-4e47-bc11-5d2659845a86" containerName="istio-proxy"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839846    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="142fff76-0542-40d8-bea6-f46a50366b1d" containerName="maas-api"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839856    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="17cf01ad-84ee-4f36-92f0-7cfff29f4215" containerName="istio-proxy"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839863    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="96eb0e3a-5bb9-45ee-ac86-80c7a0cb9c08" containerName="maas-api"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839872    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4ec0ec0-354e-41f9-897e-ecbd08404b2e" containerName="istio-proxy"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839881    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="20246613-3c91-4fc6-9c96-0e0b81709958" containerName="maas-api"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839888    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e900ea6-5bdb-40c7-bcd2-de0a6910aac0" containerName="payload-processing"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839897    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="e05329a8-9320-47fe-9766-622ae3c93bc4" containerName="maas-api"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839906    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="54b0d7fa-4b81-49a9-9819-2ee76bd5177e" containerName="istio-proxy"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839914    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5587cde-1301-458b-80c5-5a83682b3e21" containerName="istio-proxy"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839923    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde6ef20-18b2-4615-952e-8de495f731fe" containerName="istio-proxy"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839932    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="70ea964b-efdb-4d99-b4f5-e3cde4331254" containerName="main"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839941    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="245deabe-bbd0-42ef-b7dd-c264c38acdb8" containerName="maas-api"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839950    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5587cde-1301-458b-80c5-5a83682b3e21" containerName="istio-proxy"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839958    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea64a87b-85b1-46fa-9c51-4e45543bf808" containerName="maas-api"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839966    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1774f86-5e25-4358-b250-d26c6ea6c601" containerName="payload-processing"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839976    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="a54bdfac-68a5-4f09-bc41-fd51c3209245" containerName="payload-processing"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839984    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="beb04a4a-dc5c-4583-a464-3c56374faa33" containerName="payload-pre-processing"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.839991    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="47a05334-0a58-4dd9-a9e3-2194031456cd" containerName="main"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.840001    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="653f8461-4436-4506-bc3e-0271173d425e" containerName="payload-processing"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.840009    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d278356-d7d2-42cd-bba4-e5ef1867fa1c" containerName="maas-api"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.840018    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="9be3939f-afda-4003-bd81-55e3349fdd86" containerName="payload-pre-processing"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.840028    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ac6eaed-819d-4ce2-b299-b7228312da81" containerName="payload-pre-processing"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.840036    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9c8bf07-6e78-4a49-a32c-302fe418d7cc" containerName="payload-processing"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.840043    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="255ff92e-6bed-464a-a995-04cbc315cf99" containerName="maas-api"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.840052    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="734ec4d9-3542-4a3b-ba5b-d47f3ddf048e" containerName="main"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.840060    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="833c472c-0226-4e47-bc11-5d2659845a86" containerName="istio-proxy"
Oct 30 05:15:48.843020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.840070    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="d363d951-949c-4438-8c54-e65908025cfd" containerName="main"
Oct 30 05:15:48.844689 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.843319    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb"
Oct 30 05:15:48.870009 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.869985    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a566022b-dockercfg-4mfjz\""
Oct 30 05:15:48.870162 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.870008    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a566022b\""
Oct 30 05:15:48.871419 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.871396    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb"]
Oct 30 05:15:48.949115 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.949084    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfbqd\" (UniqueName: \"kubernetes.io/projected/8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd-kube-api-access-dfbqd\") pod \"payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb\" (UID: \"8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb"
Oct 30 05:15:48.949312 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:48.949126    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb\" (UID: \"8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb"
Oct 30 05:15:49.050398 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:49.050365    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfbqd\" (UniqueName: \"kubernetes.io/projected/8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd-kube-api-access-dfbqd\") pod \"payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb\" (UID: \"8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb"
Oct 30 05:15:49.050559 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:49.050407    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb\" (UID: \"8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb"
Oct 30 05:15:49.051076 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:49.051058    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb\" (UID: \"8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb"
Oct 30 05:15:49.072721 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:49.072695    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfbqd\" (UniqueName: \"kubernetes.io/projected/8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd-kube-api-access-dfbqd\") pod \"payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb\" (UID: \"8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb"
Oct 30 05:15:49.155822 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:49.155751    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb"
Oct 30 05:15:49.452899 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:49.452823    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb"]
Oct 30 05:15:49.456067 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:15:49.456020    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed4bf69_86ac_46f5_8d36_ece95b6ca1cd.slice/crio-a62c509f39218b57f95a35822c39bca01ebe32a2b5b7ea2ec4352972dff43ed4 WatchSource:0}: Error finding container a62c509f39218b57f95a35822c39bca01ebe32a2b5b7ea2ec4352972dff43ed4: Status 404 returned error can't find the container with id a62c509f39218b57f95a35822c39bca01ebe32a2b5b7ea2ec4352972dff43ed4
Oct 30 05:15:50.309063 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:50.309030    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb" event={"ID":"8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd","Type":"ContainerStarted","Data":"323dca081c2df5b1b6eccd9bcf4a51161313b1a9b398fb93624c0fa15ee2d0e9"}
Oct 30 05:15:50.309478 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:50.309079    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb" event={"ID":"8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd","Type":"ContainerStarted","Data":"a62c509f39218b57f95a35822c39bca01ebe32a2b5b7ea2ec4352972dff43ed4"}
Oct 30 05:15:50.309478 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:50.309158    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb"
Oct 30 05:15:50.347121 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:50.347077    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb" podStartSLOduration=2.347066251 podStartE2EDuration="2.347066251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:50.346523123 +0000 UTC m=+4669.758918696" watchObservedRunningTime="2026-10-30 05:15:50.347066251 +0000 UTC m=+4669.759461821"
Oct 30 05:15:53.691871 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:15:53.691831    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:15:55.608879 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:15:55.608840    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:16:01.316073 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:01.316047    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb"
Oct 30 05:16:02.875527 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:02.875496    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw"]
Oct 30 05:16:02.875923 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:02.875884    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="17cf01ad-84ee-4f36-92f0-7cfff29f4215" containerName="istio-proxy"
Oct 30 05:16:02.880520 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:02.880501    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw"
Oct 30 05:16:02.897232 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:02.897209    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\""
Oct 30 05:16:02.897232 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:02.897211    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:16:02.898922 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:02.898899    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw"]
Oct 30 05:16:02.970777 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:02.970751    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a566022b-kz6dw\" (UID: \"fcbe9987-1bce-43a9-ba8a-0f0784e531a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw"
Oct 30 05:16:02.970924 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:02.970788    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zpfp\" (UniqueName: \"kubernetes.io/projected/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-kube-api-access-4zpfp\") pod \"maas-api-revoke-keys-e2e-ait-a566022b-kz6dw\" (UID: \"fcbe9987-1bce-43a9-ba8a-0f0784e531a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw"
Oct 30 05:16:02.970924 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:02.970897    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a566022b-kz6dw\" (UID: \"fcbe9987-1bce-43a9-ba8a-0f0784e531a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw"
Oct 30 05:16:03.071824 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:03.071773    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a566022b-kz6dw\" (UID: \"fcbe9987-1bce-43a9-ba8a-0f0784e531a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw"
Oct 30 05:16:03.071974 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:03.071846    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a566022b-kz6dw\" (UID: \"fcbe9987-1bce-43a9-ba8a-0f0784e531a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw"
Oct 30 05:16:03.071974 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:03.071884    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zpfp\" (UniqueName: \"kubernetes.io/projected/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-kube-api-access-4zpfp\") pod \"maas-api-revoke-keys-e2e-ait-a566022b-kz6dw\" (UID: \"fcbe9987-1bce-43a9-ba8a-0f0784e531a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw"
Oct 30 05:16:03.072493 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:03.072472    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a566022b-kz6dw\" (UID: \"fcbe9987-1bce-43a9-ba8a-0f0784e531a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw"
Oct 30 05:16:03.074083 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:03.074061    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a566022b-kz6dw\" (UID: \"fcbe9987-1bce-43a9-ba8a-0f0784e531a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw"
Oct 30 05:16:03.098509 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:16:03.098488    4874 projected.go:196] Error preparing data for projected volume kube-api-access-4zpfp for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 30 05:16:03.098624 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:16:03.098557    4874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-kube-api-access-4zpfp podName:fcbe9987-1bce-43a9-ba8a-0f0784e531a7 nodeName:}" failed. No retries permitted until 2026-10-30 05:16:03.598542847 +0000 UTC m=+4683.010938398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4zpfp" (UniqueName: "kubernetes.io/projected/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-kube-api-access-4zpfp") pod "maas-api-revoke-keys-e2e-ait-a566022b-kz6dw" (UID: "fcbe9987-1bce-43a9-ba8a-0f0784e531a7") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 30 05:16:03.677038 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:03.676989    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zpfp\" (UniqueName: \"kubernetes.io/projected/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-kube-api-access-4zpfp\") pod \"maas-api-revoke-keys-e2e-ait-a566022b-kz6dw\" (UID: \"fcbe9987-1bce-43a9-ba8a-0f0784e531a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw"
Oct 30 05:16:03.691283 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:16:03.691257    4874 projected.go:196] Error preparing data for projected volume kube-api-access-4zpfp for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 30 05:16:03.691406 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:16:03.691331    4874 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-kube-api-access-4zpfp podName:fcbe9987-1bce-43a9-ba8a-0f0784e531a7 nodeName:}" failed. No retries permitted until 2026-10-30 05:16:04.691316622 +0000 UTC m=+4684.103712176 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4zpfp" (UniqueName: "kubernetes.io/projected/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-kube-api-access-4zpfp") pod "maas-api-revoke-keys-e2e-ait-a566022b-kz6dw" (UID: "fcbe9987-1bce-43a9-ba8a-0f0784e531a7") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 30 05:16:04.788389 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:04.788358    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zpfp\" (UniqueName: \"kubernetes.io/projected/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-kube-api-access-4zpfp\") pod \"maas-api-revoke-keys-e2e-ait-a566022b-kz6dw\" (UID: \"fcbe9987-1bce-43a9-ba8a-0f0784e531a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw"
Oct 30 05:16:04.810876 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:04.810849    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zpfp\" (UniqueName: \"kubernetes.io/projected/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-kube-api-access-4zpfp\") pod \"maas-api-revoke-keys-e2e-ait-a566022b-kz6dw\" (UID: \"fcbe9987-1bce-43a9-ba8a-0f0784e531a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw"
Oct 30 05:16:04.992611 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:04.992575    4874 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw" secret="" err="secret \"maas-api-cleanup-dockercfg-pt6g2\" not found"
Oct 30 05:16:04.992777 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:04.992656    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw"
Oct 30 05:16:05.290489 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:05.290365    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw"]
Oct 30 05:16:05.293215 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:16:05.293189    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcbe9987_1bce_43a9_ba8a_0f0784e531a7.slice/crio-49d9c97c8bfc54b057b028aa197a76719e3488325439aaefed2be26c484071e1 WatchSource:0}: Error finding container 49d9c97c8bfc54b057b028aa197a76719e3488325439aaefed2be26c484071e1: Status 404 returned error can't find the container with id 49d9c97c8bfc54b057b028aa197a76719e3488325439aaefed2be26c484071e1
Oct 30 05:16:05.376128 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:05.376098    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw" event={"ID":"fcbe9987-1bce-43a9-ba8a-0f0784e531a7","Type":"ContainerStarted","Data":"49d9c97c8bfc54b057b028aa197a76719e3488325439aaefed2be26c484071e1"}
Oct 30 05:16:06.381334 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:06.381220    4874 generic.go:356] "Generic (PLEG): container finished" podID="fcbe9987-1bce-43a9-ba8a-0f0784e531a7" containerID="60453f38396a3e5a36e7e7dcc8ab6f4043e52bd5389b6bb652f2dd8457e9f919" exitCode=0
Oct 30 05:16:06.381334 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:06.381302    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw" event={"ID":"fcbe9987-1bce-43a9-ba8a-0f0784e531a7","Type":"ContainerDied","Data":"60453f38396a3e5a36e7e7dcc8ab6f4043e52bd5389b6bb652f2dd8457e9f919"}
Oct 30 05:16:07.511701 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:07.511677    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw"
Oct 30 05:16:07.613307 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:07.613279    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-maas-api-xks-ca\") pod \"fcbe9987-1bce-43a9-ba8a-0f0784e531a7\" (UID: \"fcbe9987-1bce-43a9-ba8a-0f0784e531a7\") "
Oct 30 05:16:07.613528 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:07.613314    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-kube-api-access-4zpfp\" (UniqueName: \"kubernetes.io/projected/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-kube-api-access-4zpfp\") pod \"fcbe9987-1bce-43a9-ba8a-0f0784e531a7\" (UID: \"fcbe9987-1bce-43a9-ba8a-0f0784e531a7\") "
Oct 30 05:16:07.613528 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:07.613352    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-maas-api-openshift-ca\") pod \"fcbe9987-1bce-43a9-ba8a-0f0784e531a7\" (UID: \"fcbe9987-1bce-43a9-ba8a-0f0784e531a7\") "
Oct 30 05:16:07.613714 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:07.613691    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-maas-api-openshift-ca" pod "fcbe9987-1bce-43a9-ba8a-0f0784e531a7" (UID: "fcbe9987-1bce-43a9-ba8a-0f0784e531a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:16:07.615283 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:07.615263    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-maas-api-xks-ca" pod "fcbe9987-1bce-43a9-ba8a-0f0784e531a7" (UID: "fcbe9987-1bce-43a9-ba8a-0f0784e531a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:16:07.615400 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:07.615299    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-kube-api-access-4zpfp" pod "fcbe9987-1bce-43a9-ba8a-0f0784e531a7" (UID: "fcbe9987-1bce-43a9-ba8a-0f0784e531a7"). InnerVolumeSpecName "kube-api-access-4zpfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:16:07.714562 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:07.714501    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-maas-api-xks-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:16:07.714562 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:07.714525    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zpfp\" (UniqueName: \"kubernetes.io/projected/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-kube-api-access-4zpfp\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:16:07.714562 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:07.714534    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcbe9987-1bce-43a9-ba8a-0f0784e531a7-maas-api-openshift-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:16:08.391141 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:08.391112    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw"
Oct 30 05:16:08.391324 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:08.391105    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw" event={"ID":"fcbe9987-1bce-43a9-ba8a-0f0784e531a7","Type":"ContainerDied","Data":"49d9c97c8bfc54b057b028aa197a76719e3488325439aaefed2be26c484071e1"}
Oct 30 05:16:08.391324 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:08.391256    4874 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49d9c97c8bfc54b057b028aa197a76719e3488325439aaefed2be26c484071e1"
Oct 30 05:16:13.340496 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:13.340461    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb"]
Oct 30 05:16:16.375741 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:16.375702    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw"]
Oct 30 05:16:16.386815 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:16.386786    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a566022b-kz6dw"]
Oct 30 05:16:16.726856 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:16.726826    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcbe9987-1bce-43a9-ba8a-0f0784e531a7" path="/var/lib/kubelet/pods/fcbe9987-1bce-43a9-ba8a-0f0784e531a7/volumes"
Oct 30 05:16:17.830840 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:17.830809    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:16:18.340796 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:18.340750    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb" podUID="8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd" containerName="payload-pre-processing" containerID="cri-o://323dca081c2df5b1b6eccd9bcf4a51161313b1a9b398fb93624c0fa15ee2d0e9" gracePeriod=25
Oct 30 05:16:18.584601 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:18.584575    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb"
Oct 30 05:16:18.612753 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:18.612687    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd-plugins-config-volume\") pod \"8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd\" (UID: \"8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd\") "
Oct 30 05:16:18.612897 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:18.612826    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd-kube-api-access-dfbqd\" (UniqueName: \"kubernetes.io/projected/8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd-kube-api-access-dfbqd\") pod \"8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd\" (UID: \"8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd\") "
Oct 30 05:16:18.613078 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:18.613053    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd-plugins-config-volume" pod "8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd" (UID: "8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:16:18.614747 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:18.614727    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd-kube-api-access-dfbqd" pod "8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd" (UID: "8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd"). InnerVolumeSpecName "kube-api-access-dfbqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:16:18.714111 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:18.714080    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfbqd\" (UniqueName: \"kubernetes.io/projected/8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd-kube-api-access-dfbqd\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:16:18.714111 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:18.714110    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:16:19.441311 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:19.441275    4874 generic.go:356] "Generic (PLEG): container finished" podID="8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd" containerID="323dca081c2df5b1b6eccd9bcf4a51161313b1a9b398fb93624c0fa15ee2d0e9" exitCode=0
Oct 30 05:16:19.441311 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:19.441316    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb" event={"ID":"8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd","Type":"ContainerDied","Data":"323dca081c2df5b1b6eccd9bcf4a51161313b1a9b398fb93624c0fa15ee2d0e9"}
Oct 30 05:16:19.441977 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:19.441363    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb" event={"ID":"8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd","Type":"ContainerDied","Data":"a62c509f39218b57f95a35822c39bca01ebe32a2b5b7ea2ec4352972dff43ed4"}
Oct 30 05:16:19.441977 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:19.441373    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb"
Oct 30 05:16:19.441977 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:19.441386    4874 scope.go:122] "RemoveContainer" containerID="323dca081c2df5b1b6eccd9bcf4a51161313b1a9b398fb93624c0fa15ee2d0e9"
Oct 30 05:16:19.449442 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:19.449424    4874 scope.go:122] "RemoveContainer" containerID="323dca081c2df5b1b6eccd9bcf4a51161313b1a9b398fb93624c0fa15ee2d0e9"
Oct 30 05:16:19.449699 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:16:19.449672    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"323dca081c2df5b1b6eccd9bcf4a51161313b1a9b398fb93624c0fa15ee2d0e9\": container with ID starting with 323dca081c2df5b1b6eccd9bcf4a51161313b1a9b398fb93624c0fa15ee2d0e9 not found: ID does not exist" containerID="323dca081c2df5b1b6eccd9bcf4a51161313b1a9b398fb93624c0fa15ee2d0e9"
Oct 30 05:16:19.449765 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:19.449710    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"323dca081c2df5b1b6eccd9bcf4a51161313b1a9b398fb93624c0fa15ee2d0e9"} err="failed to get container status \"323dca081c2df5b1b6eccd9bcf4a51161313b1a9b398fb93624c0fa15ee2d0e9\": rpc error: code = NotFound desc = could not find container \"323dca081c2df5b1b6eccd9bcf4a51161313b1a9b398fb93624c0fa15ee2d0e9\": container with ID starting with 323dca081c2df5b1b6eccd9bcf4a51161313b1a9b398fb93624c0fa15ee2d0e9 not found: ID does not exist"
Oct 30 05:16:19.490875 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:19.490852    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb"]
Oct 30 05:16:19.501788 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:19.501769    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a566022b-c898dff55-t8pdb"]
Oct 30 05:16:20.733391 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:20.733360    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd" path="/var/lib/kubelet/pods/8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd/volumes"
Oct 30 05:16:25.654616 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:16:25.654581    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:16:46.762592 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:46.762560    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m25pc"]
Oct 30 05:16:46.763025 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:46.762852    4874 cpu_manager.go:414] "containerMap: removing container" podUID="8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd" containerName="payload-pre-processing"
Oct 30 05:16:46.763025 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:46.762863    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd" containerName="payload-pre-processing"
Oct 30 05:16:46.763025 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:46.762875    4874 cpu_manager.go:414] "containerMap: removing container" podUID="fcbe9987-1bce-43a9-ba8a-0f0784e531a7" containerName="revoke-keys"
Oct 30 05:16:46.763025 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:46.762880    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcbe9987-1bce-43a9-ba8a-0f0784e531a7" containerName="revoke-keys"
Oct 30 05:16:46.763025 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:46.762943    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcbe9987-1bce-43a9-ba8a-0f0784e531a7" containerName="revoke-keys"
Oct 30 05:16:46.763025 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:46.762957    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ed4bf69-86ac-46f5-8d36-ece95b6ca1cd" containerName="payload-pre-processing"
Oct 30 05:16:46.765749 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:46.765728    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m25pc"
Oct 30 05:16:46.779530 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:46.779512    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:16:46.785364 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:46.785341    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d6a1216-a804-43f9-b5bf-60d9c511df94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-m25pc\" (UID: \"2d6a1216-a804-43f9-b5bf-60d9c511df94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m25pc"
Oct 30 05:16:46.785553 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:46.785528    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d6a1216-a804-43f9-b5bf-60d9c511df94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-m25pc\" (UID: \"2d6a1216-a804-43f9-b5bf-60d9c511df94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m25pc"
Oct 30 05:16:46.785611 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:46.785596    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw76b\" (UniqueName: \"kubernetes.io/projected/2d6a1216-a804-43f9-b5bf-60d9c511df94-kube-api-access-rw76b\") pod \"maas-api-revoke-sub-85237e45-m25pc\" (UID: \"2d6a1216-a804-43f9-b5bf-60d9c511df94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m25pc"
Oct 30 05:16:46.790716 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:46.790695    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:16:46.792901 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:46.792878    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m25pc"]
Oct 30 05:16:46.886497 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:46.886469    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d6a1216-a804-43f9-b5bf-60d9c511df94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-m25pc\" (UID: \"2d6a1216-a804-43f9-b5bf-60d9c511df94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m25pc"
Oct 30 05:16:46.886664 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:46.886508    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw76b\" (UniqueName: \"kubernetes.io/projected/2d6a1216-a804-43f9-b5bf-60d9c511df94-kube-api-access-rw76b\") pod \"maas-api-revoke-sub-85237e45-m25pc\" (UID: \"2d6a1216-a804-43f9-b5bf-60d9c511df94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m25pc"
Oct 30 05:16:46.886664 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:46.886626    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d6a1216-a804-43f9-b5bf-60d9c511df94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-m25pc\" (UID: \"2d6a1216-a804-43f9-b5bf-60d9c511df94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m25pc"
Oct 30 05:16:46.887159 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:46.887133    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d6a1216-a804-43f9-b5bf-60d9c511df94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-m25pc\" (UID: \"2d6a1216-a804-43f9-b5bf-60d9c511df94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m25pc"
Oct 30 05:16:46.888867 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:46.888847    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d6a1216-a804-43f9-b5bf-60d9c511df94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-m25pc\" (UID: \"2d6a1216-a804-43f9-b5bf-60d9c511df94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m25pc"
Oct 30 05:16:46.908175 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:46.908150    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw76b\" (UniqueName: \"kubernetes.io/projected/2d6a1216-a804-43f9-b5bf-60d9c511df94-kube-api-access-rw76b\") pod \"maas-api-revoke-sub-85237e45-m25pc\" (UID: \"2d6a1216-a804-43f9-b5bf-60d9c511df94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m25pc"
Oct 30 05:16:47.077766 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:47.077696    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m25pc"
Oct 30 05:16:47.372861 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:47.372835    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m25pc"]
Oct 30 05:16:47.375304 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:16:47.375274    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6a1216_a804_43f9_b5bf_60d9c511df94.slice/crio-fca996a12399b3710d0c34b08d191c8f7bbdf8e6c56ef3e37e602564ded52d49 WatchSource:0}: Error finding container fca996a12399b3710d0c34b08d191c8f7bbdf8e6c56ef3e37e602564ded52d49: Status 404 returned error can't find the container with id fca996a12399b3710d0c34b08d191c8f7bbdf8e6c56ef3e37e602564ded52d49
Oct 30 05:16:47.572220 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:47.572188    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m25pc" event={"ID":"2d6a1216-a804-43f9-b5bf-60d9c511df94","Type":"ContainerStarted","Data":"fca996a12399b3710d0c34b08d191c8f7bbdf8e6c56ef3e37e602564ded52d49"}
Oct 30 05:16:48.580339 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:48.580300    4874 generic.go:356] "Generic (PLEG): container finished" podID="2d6a1216-a804-43f9-b5bf-60d9c511df94" containerID="cd75a6e9e06218b70e64668d1ed3dfe04ca62b5cc743a009b6fe3e47e4541f43" exitCode=0
Oct 30 05:16:48.580800 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:48.580386    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m25pc" event={"ID":"2d6a1216-a804-43f9-b5bf-60d9c511df94","Type":"ContainerDied","Data":"cd75a6e9e06218b70e64668d1ed3dfe04ca62b5cc743a009b6fe3e47e4541f43"}
Oct 30 05:16:49.708444 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:49.708420    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m25pc"
Oct 30 05:16:49.810780 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:49.810744    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d6a1216-a804-43f9-b5bf-60d9c511df94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d6a1216-a804-43f9-b5bf-60d9c511df94-maas-api-openshift-ca\") pod \"2d6a1216-a804-43f9-b5bf-60d9c511df94\" (UID: \"2d6a1216-a804-43f9-b5bf-60d9c511df94\") "
Oct 30 05:16:49.811463 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:49.811433    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d6a1216-a804-43f9-b5bf-60d9c511df94-kube-api-access-rw76b\" (UniqueName: \"kubernetes.io/projected/2d6a1216-a804-43f9-b5bf-60d9c511df94-kube-api-access-rw76b\") pod \"2d6a1216-a804-43f9-b5bf-60d9c511df94\" (UID: \"2d6a1216-a804-43f9-b5bf-60d9c511df94\") "
Oct 30 05:16:49.811941 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:49.811341    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d6a1216-a804-43f9-b5bf-60d9c511df94-maas-api-openshift-ca" pod "2d6a1216-a804-43f9-b5bf-60d9c511df94" (UID: "2d6a1216-a804-43f9-b5bf-60d9c511df94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:16:49.812072 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:49.811961    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d6a1216-a804-43f9-b5bf-60d9c511df94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d6a1216-a804-43f9-b5bf-60d9c511df94-maas-api-xks-ca\") pod \"2d6a1216-a804-43f9-b5bf-60d9c511df94\" (UID: \"2d6a1216-a804-43f9-b5bf-60d9c511df94\") "
Oct 30 05:16:49.812317 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:49.812298    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d6a1216-a804-43f9-b5bf-60d9c511df94-maas-api-openshift-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:16:49.813983 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:49.813956    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d6a1216-a804-43f9-b5bf-60d9c511df94-maas-api-xks-ca" pod "2d6a1216-a804-43f9-b5bf-60d9c511df94" (UID: "2d6a1216-a804-43f9-b5bf-60d9c511df94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:16:49.814599 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:49.814572    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d6a1216-a804-43f9-b5bf-60d9c511df94-kube-api-access-rw76b" pod "2d6a1216-a804-43f9-b5bf-60d9c511df94" (UID: "2d6a1216-a804-43f9-b5bf-60d9c511df94"). InnerVolumeSpecName "kube-api-access-rw76b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:16:49.913043 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:49.912956    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d6a1216-a804-43f9-b5bf-60d9c511df94-maas-api-xks-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:16:49.913043 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:49.912990    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw76b\" (UniqueName: \"kubernetes.io/projected/2d6a1216-a804-43f9-b5bf-60d9c511df94-kube-api-access-rw76b\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:16:50.589442 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:50.589414    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m25pc"
Oct 30 05:16:50.589442 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:50.589425    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m25pc" event={"ID":"2d6a1216-a804-43f9-b5bf-60d9c511df94","Type":"ContainerDied","Data":"fca996a12399b3710d0c34b08d191c8f7bbdf8e6c56ef3e37e602564ded52d49"}
Oct 30 05:16:50.589654 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:16:50.589466    4874 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fca996a12399b3710d0c34b08d191c8f7bbdf8e6c56ef3e37e602564ded52d49"
Oct 30 05:16:55.607882 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:16:55.607844    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:17:01.155329 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:01.155297    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f4559f84c-tdxj7"]
Oct 30 05:17:01.155728 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:01.155622    4874 cpu_manager.go:414] "containerMap: removing container" podUID="2d6a1216-a804-43f9-b5bf-60d9c511df94" containerName="revoke-keys"
Oct 30 05:17:01.155728 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:01.155633    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d6a1216-a804-43f9-b5bf-60d9c511df94" containerName="revoke-keys"
Oct 30 05:17:01.155728 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:01.155686    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d6a1216-a804-43f9-b5bf-60d9c511df94" containerName="revoke-keys"
Oct 30 05:17:01.158764 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:01.158746    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f4559f84c-tdxj7"
Oct 30 05:17:01.175642 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:01.175612    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 05:17:01.186570 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:01.186544    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 05:17:01.186757 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:01.186742    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 05:17:01.186757 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:01.186742    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j8pc2\""
Oct 30 05:17:01.186854 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:01.186784    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 05:17:01.186854 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:01.186788    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 05:17:01.188541 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:01.188517    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f4559f84c-tdxj7"]
Oct 30 05:17:01.214089 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:01.214067    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24d61891-9970-444b-9a0d-80a7e911c420-metrics-certs\") pod \"maas-controller-6f4559f84c-tdxj7\" (UID: \"24d61891-9970-444b-9a0d-80a7e911c420\") " pod="opendatahub/maas-controller-6f4559f84c-tdxj7"
Oct 30 05:17:01.214207 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:01.214108    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t2df\" (UniqueName: \"kubernetes.io/projected/24d61891-9970-444b-9a0d-80a7e911c420-kube-api-access-7t2df\") pod \"maas-controller-6f4559f84c-tdxj7\" (UID: \"24d61891-9970-444b-9a0d-80a7e911c420\") " pod="opendatahub/maas-controller-6f4559f84c-tdxj7"
Oct 30 05:17:01.214207 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:01.214126    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24d61891-9970-444b-9a0d-80a7e911c420-webhook-cert\") pod \"maas-controller-6f4559f84c-tdxj7\" (UID: \"24d61891-9970-444b-9a0d-80a7e911c420\") " pod="opendatahub/maas-controller-6f4559f84c-tdxj7"
Oct 30 05:17:01.314925 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:01.314903    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24d61891-9970-444b-9a0d-80a7e911c420-metrics-certs\") pod \"maas-controller-6f4559f84c-tdxj7\" (UID: \"24d61891-9970-444b-9a0d-80a7e911c420\") " pod="opendatahub/maas-controller-6f4559f84c-tdxj7"
Oct 30 05:17:01.315113 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:01.314955    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t2df\" (UniqueName: \"kubernetes.io/projected/24d61891-9970-444b-9a0d-80a7e911c420-kube-api-access-7t2df\") pod \"maas-controller-6f4559f84c-tdxj7\" (UID: \"24d61891-9970-444b-9a0d-80a7e911c420\") " pod="opendatahub/maas-controller-6f4559f84c-tdxj7"
Oct 30 05:17:01.315113 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:01.314979    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24d61891-9970-444b-9a0d-80a7e911c420-webhook-cert\") pod \"maas-controller-6f4559f84c-tdxj7\" (UID: \"24d61891-9970-444b-9a0d-80a7e911c420\") " pod="opendatahub/maas-controller-6f4559f84c-tdxj7"
Oct 30 05:17:01.317474 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:01.317452    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24d61891-9970-444b-9a0d-80a7e911c420-metrics-certs\") pod \"maas-controller-6f4559f84c-tdxj7\" (UID: \"24d61891-9970-444b-9a0d-80a7e911c420\") " pod="opendatahub/maas-controller-6f4559f84c-tdxj7"
Oct 30 05:17:01.317474 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:01.317470    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24d61891-9970-444b-9a0d-80a7e911c420-webhook-cert\") pod \"maas-controller-6f4559f84c-tdxj7\" (UID: \"24d61891-9970-444b-9a0d-80a7e911c420\") " pod="opendatahub/maas-controller-6f4559f84c-tdxj7"
Oct 30 05:17:01.337014 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:01.336991    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t2df\" (UniqueName: \"kubernetes.io/projected/24d61891-9970-444b-9a0d-80a7e911c420-kube-api-access-7t2df\") pod \"maas-controller-6f4559f84c-tdxj7\" (UID: \"24d61891-9970-444b-9a0d-80a7e911c420\") " pod="opendatahub/maas-controller-6f4559f84c-tdxj7"
Oct 30 05:17:01.472206 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:01.472125    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f4559f84c-tdxj7"
Oct 30 05:17:01.768551 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:01.768515    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f4559f84c-tdxj7"]
Oct 30 05:17:01.771385 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:17:01.771359    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d61891_9970_444b_9a0d_80a7e911c420.slice/crio-78aa7ca298b0f4e1c22f60fb5f6b997d8ffee1c637789f4e8fa4155c3f011eb4 WatchSource:0}: Error finding container 78aa7ca298b0f4e1c22f60fb5f6b997d8ffee1c637789f4e8fa4155c3f011eb4: Status 404 returned error can't find the container with id 78aa7ca298b0f4e1c22f60fb5f6b997d8ffee1c637789f4e8fa4155c3f011eb4
Oct 30 05:17:01.772591 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:01.772573    4874 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 05:17:02.640342 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:02.640263    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f4559f84c-tdxj7" event={"ID":"24d61891-9970-444b-9a0d-80a7e911c420","Type":"ContainerStarted","Data":"78aa7ca298b0f4e1c22f60fb5f6b997d8ffee1c637789f4e8fa4155c3f011eb4"}
Oct 30 05:17:03.645829 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:03.645797    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f4559f84c-tdxj7" event={"ID":"24d61891-9970-444b-9a0d-80a7e911c420","Type":"ContainerStarted","Data":"2b009a07be6930671b86231cefd0b58a39a17034594210b4e86eb57c6a424c48"}
Oct 30 05:17:03.646351 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:03.645933    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f4559f84c-tdxj7"
Oct 30 05:17:03.688377 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:03.688320    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6f4559f84c-tdxj7" podStartSLOduration=1.296086177 podStartE2EDuration="2.688307116s" totalImagesPullingTime="1.392220939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:17:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:17:03.686909471 +0000 UTC m=+4743.099305043" watchObservedRunningTime="2026-10-30 05:17:03.688307116 +0000 UTC m=+4743.100702687"
Oct 30 05:17:14.652586 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:14.652549    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f4559f84c-tdxj7"
Oct 30 05:17:23.411767 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:23.411731    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:17:25.653189 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:17:25.653148    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:17:39.410860 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:39.410818    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:17:44.086009 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.085971    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"]
Oct 30 05:17:44.100143 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.100118    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.121260 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.119801    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/805efc98-2476-40e3-9613-69336ac75dca-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.121260 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.119854    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/805efc98-2476-40e3-9613-69336ac75dca-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.121260 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.119894    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/805efc98-2476-40e3-9613-69336ac75dca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.121260 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.119930    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/805efc98-2476-40e3-9613-69336ac75dca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.121260 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.119955    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/805efc98-2476-40e3-9613-69336ac75dca-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.121260 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.119988    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/805efc98-2476-40e3-9613-69336ac75dca-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.121260 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.120024    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7knnt\" (UniqueName: \"kubernetes.io/projected/805efc98-2476-40e3-9613-69336ac75dca-kube-api-access-7knnt\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.121260 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.120092    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/805efc98-2476-40e3-9613-69336ac75dca-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.121260 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.120137    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/805efc98-2476-40e3-9613-69336ac75dca-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.150054 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.150020    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"]
Oct 30 05:17:44.222832 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.222788    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/805efc98-2476-40e3-9613-69336ac75dca-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.223131 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.223093    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/805efc98-2476-40e3-9613-69336ac75dca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.227398 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.227352    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/805efc98-2476-40e3-9613-69336ac75dca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.227583 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.227561    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/805efc98-2476-40e3-9613-69336ac75dca-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.227749 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.227731    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/805efc98-2476-40e3-9613-69336ac75dca-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.227909 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.227891    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7knnt\" (UniqueName: \"kubernetes.io/projected/805efc98-2476-40e3-9613-69336ac75dca-kube-api-access-7knnt\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.228112 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.228096    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/805efc98-2476-40e3-9613-69336ac75dca-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.229028 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.228947    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/805efc98-2476-40e3-9613-69336ac75dca-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.237297 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.232330    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/805efc98-2476-40e3-9613-69336ac75dca-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.237297 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.232433    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/805efc98-2476-40e3-9613-69336ac75dca-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.237297 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.232849    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/805efc98-2476-40e3-9613-69336ac75dca-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.237297 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.233166    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/805efc98-2476-40e3-9613-69336ac75dca-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.237297 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.236952    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/805efc98-2476-40e3-9613-69336ac75dca-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.238386 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.238363    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/805efc98-2476-40e3-9613-69336ac75dca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.243820 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.243792    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/805efc98-2476-40e3-9613-69336ac75dca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.244569 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.244538    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/805efc98-2476-40e3-9613-69336ac75dca-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.250495 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.250475    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/805efc98-2476-40e3-9613-69336ac75dca-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.262861 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.262842    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7knnt\" (UniqueName: \"kubernetes.io/projected/805efc98-2476-40e3-9613-69336ac75dca-kube-api-access-7knnt\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz6cz\" (UID: \"805efc98-2476-40e3-9613-69336ac75dca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.444231 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.444146    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:44.837266 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.834186    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"]
Oct 30 05:17:44.837794 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:17:44.837509    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805efc98_2476_40e3_9613_69336ac75dca.slice/crio-eff8817f36b5348875eedcfb6da1ac6abb257bbdf574bf86296298f366548850 WatchSource:0}: Error finding container eff8817f36b5348875eedcfb6da1ac6abb257bbdf574bf86296298f366548850: Status 404 returned error can't find the container with id eff8817f36b5348875eedcfb6da1ac6abb257bbdf574bf86296298f366548850
Oct 30 05:17:44.842058 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.839906    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 05:17:44.842058 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.839997    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 05:17:44.842058 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:44.840039    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 05:17:45.528493 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:45.528460    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:17:45.848711 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:45.848621    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz" event={"ID":"805efc98-2476-40e3-9613-69336ac75dca","Type":"ContainerStarted","Data":"f5dd33251feb0b1715d000d331700c722f078a0e713bd8221b9f69ae8d7ebf6d"}
Oct 30 05:17:45.848711 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:45.848671    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz" event={"ID":"805efc98-2476-40e3-9613-69336ac75dca","Type":"ContainerStarted","Data":"eff8817f36b5348875eedcfb6da1ac6abb257bbdf574bf86296298f366548850"}
Oct 30 05:17:45.916425 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:45.916361    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz" podStartSLOduration=1.916343342 podStartE2EDuration="1.916343342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:17:45.914857894 +0000 UTC m=+4785.327253468" watchObservedRunningTime="2026-10-30 05:17:45.916343342 +0000 UTC m=+4785.328738917"
Oct 30 05:17:46.445334 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:46.445300    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:46.446757 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:46.446725    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hz6cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 30 05:17:46.446902 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:46.446775    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz" podUID="805efc98-2476-40e3-9613-69336ac75dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 30 05:17:47.445131 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:47.445090    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hz6cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 30 05:17:47.445665 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:47.445155    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz" podUID="805efc98-2476-40e3-9613-69336ac75dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 30 05:17:48.444611 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:48.444573    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hz6cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 30 05:17:48.444797 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:48.444639    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz" podUID="805efc98-2476-40e3-9613-69336ac75dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 30 05:17:49.445067 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:49.445026    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hz6cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 30 05:17:49.445555 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:49.445100    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz" podUID="805efc98-2476-40e3-9613-69336ac75dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 30 05:17:50.446692 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:50.446649    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hz6cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 30 05:17:50.447456 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:50.447418    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz" podUID="805efc98-2476-40e3-9613-69336ac75dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 30 05:17:51.054483 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:51.054438    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hwcsp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:17:51.054711 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:51.054501    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp" podUID="62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:17:51.154999 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:51.154963    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:17:51.444980 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:51.444883    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hz6cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 30 05:17:51.445281 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:51.445219    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz" podUID="805efc98-2476-40e3-9613-69336ac75dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 30 05:17:52.444966 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:52.444932    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hz6cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 30 05:17:52.445388 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:52.444985    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz" podUID="805efc98-2476-40e3-9613-69336ac75dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 30 05:17:54.446180 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:54.446139    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hz6cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:17:54.446656 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:54.446213    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz" podUID="805efc98-2476-40e3-9613-69336ac75dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:17:55.447505 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:55.447466    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hz6cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:17:55.447887 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:55.447528    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz" podUID="805efc98-2476-40e3-9613-69336ac75dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:17:55.451252 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:55.451213    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:17:55.615662 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:17:55.615622    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:17:55.682039 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:55.682012    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:55.682350 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:55.682327    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:17:55.683100 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:17:55.683077    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz"
Oct 30 05:18:01.410769 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:18:01.410746    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 05:18:01.415372 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:18:01.415352    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 05:18:07.778269 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:18:07.777394    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-x8mpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:18:07.778269 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:18:07.777477    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd" podUID="c2ea846a-5ce0-424a-bb80-ec759b592c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:18:07.794405 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:18:07.794370    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fh868 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:18:07.794668 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:18:07.794639    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868" podUID="7059d28d-3eca-44fa-80c4-ac634a59aaf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:18:25.608784 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:18:25.608702    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:18:31.029330 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:18:31.029293    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:18:37.787264 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:18:37.786329    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fh868 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:18:37.787264 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:18:37.786410    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868" podUID="7059d28d-3eca-44fa-80c4-ac634a59aaf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:18:39.204817 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:18:39.204785    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:18:55.608548 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:18:55.608509    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:19:15.417197 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:15.417160    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:19:22.905583 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:22.905550    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a00ef9d-s5svp"]
Oct 30 05:19:22.908899 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:22.908877    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a00ef9d-s5svp"
Oct 30 05:19:22.925261 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:22.925216    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:19:22.936553 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:22.936530    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:19:22.938399 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:22.938379    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a00ef9d-s5svp"]
Oct 30 05:19:22.982886 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:22.982846    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebddcf1-9702-4cd7-8931-525ca45c52ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a00ef9d-s5svp\" (UID: \"6ebddcf1-9702-4cd7-8931-525ca45c52ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a00ef9d-s5svp"
Oct 30 05:19:22.983031 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:22.982890    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jjhw\" (UniqueName: \"kubernetes.io/projected/6ebddcf1-9702-4cd7-8931-525ca45c52ed-kube-api-access-7jjhw\") pod \"maas-api-revoke-sub-7a00ef9d-s5svp\" (UID: \"6ebddcf1-9702-4cd7-8931-525ca45c52ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a00ef9d-s5svp"
Oct 30 05:19:22.983031 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:22.982999    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebddcf1-9702-4cd7-8931-525ca45c52ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a00ef9d-s5svp\" (UID: \"6ebddcf1-9702-4cd7-8931-525ca45c52ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a00ef9d-s5svp"
Oct 30 05:19:23.084145 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:23.084111    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebddcf1-9702-4cd7-8931-525ca45c52ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a00ef9d-s5svp\" (UID: \"6ebddcf1-9702-4cd7-8931-525ca45c52ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a00ef9d-s5svp"
Oct 30 05:19:23.084316 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:23.084164    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebddcf1-9702-4cd7-8931-525ca45c52ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a00ef9d-s5svp\" (UID: \"6ebddcf1-9702-4cd7-8931-525ca45c52ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a00ef9d-s5svp"
Oct 30 05:19:23.084316 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:23.084188    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jjhw\" (UniqueName: \"kubernetes.io/projected/6ebddcf1-9702-4cd7-8931-525ca45c52ed-kube-api-access-7jjhw\") pod \"maas-api-revoke-sub-7a00ef9d-s5svp\" (UID: \"6ebddcf1-9702-4cd7-8931-525ca45c52ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a00ef9d-s5svp"
Oct 30 05:19:23.084800 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:23.084775    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebddcf1-9702-4cd7-8931-525ca45c52ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a00ef9d-s5svp\" (UID: \"6ebddcf1-9702-4cd7-8931-525ca45c52ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a00ef9d-s5svp"
Oct 30 05:19:23.086557 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:23.086535    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebddcf1-9702-4cd7-8931-525ca45c52ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a00ef9d-s5svp\" (UID: \"6ebddcf1-9702-4cd7-8931-525ca45c52ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a00ef9d-s5svp"
Oct 30 05:19:23.106730 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:23.106702    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jjhw\" (UniqueName: \"kubernetes.io/projected/6ebddcf1-9702-4cd7-8931-525ca45c52ed-kube-api-access-7jjhw\") pod \"maas-api-revoke-sub-7a00ef9d-s5svp\" (UID: \"6ebddcf1-9702-4cd7-8931-525ca45c52ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a00ef9d-s5svp"
Oct 30 05:19:23.220983 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:23.220921    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a00ef9d-s5svp"
Oct 30 05:19:23.516828 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:23.516804    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a00ef9d-s5svp"]
Oct 30 05:19:23.518585 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:19:23.518557    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebddcf1_9702_4cd7_8931_525ca45c52ed.slice/crio-9dbc3b5304edaf6f08ee793c26573158e8777a1f0e206730205b46972a30cef2 WatchSource:0}: Error finding container 9dbc3b5304edaf6f08ee793c26573158e8777a1f0e206730205b46972a30cef2: Status 404 returned error can't find the container with id 9dbc3b5304edaf6f08ee793c26573158e8777a1f0e206730205b46972a30cef2
Oct 30 05:19:24.431207 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:24.431174    4874 generic.go:356] "Generic (PLEG): container finished" podID="6ebddcf1-9702-4cd7-8931-525ca45c52ed" containerID="6d4ac414f57ca0c7d6c6a1958f6d92b07143ac1cdb9dea9d2d1d17f21f6bba9d" exitCode=0
Oct 30 05:19:24.431562 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:24.431267    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a00ef9d-s5svp" event={"ID":"6ebddcf1-9702-4cd7-8931-525ca45c52ed","Type":"ContainerDied","Data":"6d4ac414f57ca0c7d6c6a1958f6d92b07143ac1cdb9dea9d2d1d17f21f6bba9d"}
Oct 30 05:19:24.431562 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:24.431319    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a00ef9d-s5svp" event={"ID":"6ebddcf1-9702-4cd7-8931-525ca45c52ed","Type":"ContainerStarted","Data":"9dbc3b5304edaf6f08ee793c26573158e8777a1f0e206730205b46972a30cef2"}
Oct 30 05:19:25.562535 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:25.562509    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a00ef9d-s5svp"
Oct 30 05:19:25.610469 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:19:25.610433    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:19:25.706867 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:25.706776    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ebddcf1-9702-4cd7-8931-525ca45c52ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebddcf1-9702-4cd7-8931-525ca45c52ed-maas-api-xks-ca\") pod \"6ebddcf1-9702-4cd7-8931-525ca45c52ed\" (UID: \"6ebddcf1-9702-4cd7-8931-525ca45c52ed\") "
Oct 30 05:19:25.707034 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:25.706940    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ebddcf1-9702-4cd7-8931-525ca45c52ed-kube-api-access-7jjhw\" (UniqueName: \"kubernetes.io/projected/6ebddcf1-9702-4cd7-8931-525ca45c52ed-kube-api-access-7jjhw\") pod \"6ebddcf1-9702-4cd7-8931-525ca45c52ed\" (UID: \"6ebddcf1-9702-4cd7-8931-525ca45c52ed\") "
Oct 30 05:19:25.707034 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:25.706977    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ebddcf1-9702-4cd7-8931-525ca45c52ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebddcf1-9702-4cd7-8931-525ca45c52ed-maas-api-openshift-ca\") pod \"6ebddcf1-9702-4cd7-8931-525ca45c52ed\" (UID: \"6ebddcf1-9702-4cd7-8931-525ca45c52ed\") "
Oct 30 05:19:25.707447 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:25.707419    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ebddcf1-9702-4cd7-8931-525ca45c52ed-maas-api-openshift-ca" pod "6ebddcf1-9702-4cd7-8931-525ca45c52ed" (UID: "6ebddcf1-9702-4cd7-8931-525ca45c52ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:19:25.708929 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:25.708905    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ebddcf1-9702-4cd7-8931-525ca45c52ed-maas-api-xks-ca" pod "6ebddcf1-9702-4cd7-8931-525ca45c52ed" (UID: "6ebddcf1-9702-4cd7-8931-525ca45c52ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:19:25.709022 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:25.708976    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebddcf1-9702-4cd7-8931-525ca45c52ed-kube-api-access-7jjhw" pod "6ebddcf1-9702-4cd7-8931-525ca45c52ed" (UID: "6ebddcf1-9702-4cd7-8931-525ca45c52ed"). InnerVolumeSpecName "kube-api-access-7jjhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:19:25.808416 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:25.808381    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jjhw\" (UniqueName: \"kubernetes.io/projected/6ebddcf1-9702-4cd7-8931-525ca45c52ed-kube-api-access-7jjhw\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:19:25.808416 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:25.808410    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebddcf1-9702-4cd7-8931-525ca45c52ed-maas-api-openshift-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:19:25.808416 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:25.808419    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebddcf1-9702-4cd7-8931-525ca45c52ed-maas-api-xks-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:19:26.441221 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:26.441178    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a00ef9d-s5svp" event={"ID":"6ebddcf1-9702-4cd7-8931-525ca45c52ed","Type":"ContainerDied","Data":"9dbc3b5304edaf6f08ee793c26573158e8777a1f0e206730205b46972a30cef2"}
Oct 30 05:19:26.441386 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:26.441230    4874 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9dbc3b5304edaf6f08ee793c26573158e8777a1f0e206730205b46972a30cef2"
Oct 30 05:19:26.441386 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:26.441194    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a00ef9d-s5svp"
Oct 30 05:19:34.527995 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:34.527962    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:19:37.661339 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:37.661308    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4f5e14b-vp9nb"]
Oct 30 05:19:37.661734 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:37.661657    4874 cpu_manager.go:414] "containerMap: removing container" podUID="6ebddcf1-9702-4cd7-8931-525ca45c52ed" containerName="revoke-keys"
Oct 30 05:19:37.661734 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:37.661669    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ebddcf1-9702-4cd7-8931-525ca45c52ed" containerName="revoke-keys"
Oct 30 05:19:37.661846 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:37.661746    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ebddcf1-9702-4cd7-8931-525ca45c52ed" containerName="revoke-keys"
Oct 30 05:19:37.665197 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:37.665171    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4f5e14b-vp9nb"
Oct 30 05:19:37.692478 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:37.692444    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:19:37.692691 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:37.692500    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:19:37.694253 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:37.694211    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4f5e14b-vp9nb"]
Oct 30 05:19:37.713500 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:37.713463    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl8dr\" (UniqueName: \"kubernetes.io/projected/a439cc07-946c-4119-bfb8-3f04130f52e8-kube-api-access-jl8dr\") pod \"maas-api-revoke-sub-f4f5e14b-vp9nb\" (UID: \"a439cc07-946c-4119-bfb8-3f04130f52e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4f5e14b-vp9nb"
Oct 30 05:19:37.713662 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:37.713611    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a439cc07-946c-4119-bfb8-3f04130f52e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4f5e14b-vp9nb\" (UID: \"a439cc07-946c-4119-bfb8-3f04130f52e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4f5e14b-vp9nb"
Oct 30 05:19:37.713755 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:37.713684    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a439cc07-946c-4119-bfb8-3f04130f52e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4f5e14b-vp9nb\" (UID: \"a439cc07-946c-4119-bfb8-3f04130f52e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4f5e14b-vp9nb"
Oct 30 05:19:37.814448 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:37.814407    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a439cc07-946c-4119-bfb8-3f04130f52e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4f5e14b-vp9nb\" (UID: \"a439cc07-946c-4119-bfb8-3f04130f52e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4f5e14b-vp9nb"
Oct 30 05:19:37.814630 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:37.814491    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a439cc07-946c-4119-bfb8-3f04130f52e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4f5e14b-vp9nb\" (UID: \"a439cc07-946c-4119-bfb8-3f04130f52e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4f5e14b-vp9nb"
Oct 30 05:19:37.814630 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:37.814589    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl8dr\" (UniqueName: \"kubernetes.io/projected/a439cc07-946c-4119-bfb8-3f04130f52e8-kube-api-access-jl8dr\") pod \"maas-api-revoke-sub-f4f5e14b-vp9nb\" (UID: \"a439cc07-946c-4119-bfb8-3f04130f52e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4f5e14b-vp9nb"
Oct 30 05:19:37.815192 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:37.815169    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a439cc07-946c-4119-bfb8-3f04130f52e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4f5e14b-vp9nb\" (UID: \"a439cc07-946c-4119-bfb8-3f04130f52e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4f5e14b-vp9nb"
Oct 30 05:19:37.816674 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:37.816653    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a439cc07-946c-4119-bfb8-3f04130f52e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4f5e14b-vp9nb\" (UID: \"a439cc07-946c-4119-bfb8-3f04130f52e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4f5e14b-vp9nb"
Oct 30 05:19:37.837866 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:37.837838    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl8dr\" (UniqueName: \"kubernetes.io/projected/a439cc07-946c-4119-bfb8-3f04130f52e8-kube-api-access-jl8dr\") pod \"maas-api-revoke-sub-f4f5e14b-vp9nb\" (UID: \"a439cc07-946c-4119-bfb8-3f04130f52e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4f5e14b-vp9nb"
Oct 30 05:19:37.980251 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:37.980214    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4f5e14b-vp9nb"
Oct 30 05:19:38.278501 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:38.278464    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4f5e14b-vp9nb"]
Oct 30 05:19:38.280556 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:19:38.280529    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda439cc07_946c_4119_bfb8_3f04130f52e8.slice/crio-6814c1d04a6e23bdaa6aa805e32eae3c396490f7829a43f7744641f46953c7dd WatchSource:0}: Error finding container 6814c1d04a6e23bdaa6aa805e32eae3c396490f7829a43f7744641f46953c7dd: Status 404 returned error can't find the container with id 6814c1d04a6e23bdaa6aa805e32eae3c396490f7829a43f7744641f46953c7dd
Oct 30 05:19:38.495695 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:38.495639    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4f5e14b-vp9nb" event={"ID":"a439cc07-946c-4119-bfb8-3f04130f52e8","Type":"ContainerStarted","Data":"6814c1d04a6e23bdaa6aa805e32eae3c396490f7829a43f7744641f46953c7dd"}
Oct 30 05:19:39.527262 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:39.526822    4874 generic.go:356] "Generic (PLEG): container finished" podID="a439cc07-946c-4119-bfb8-3f04130f52e8" containerID="fe26c0e5faf0af386ed72a148b745cd956c06493e4e4227cc08a4e8be103461e" exitCode=0
Oct 30 05:19:39.527262 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:39.526899    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4f5e14b-vp9nb" event={"ID":"a439cc07-946c-4119-bfb8-3f04130f52e8","Type":"ContainerDied","Data":"fe26c0e5faf0af386ed72a148b745cd956c06493e4e4227cc08a4e8be103461e"}
Oct 30 05:19:41.089444 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:41.089361    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4f5e14b-vp9nb"
Oct 30 05:19:41.194734 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:41.194702    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a439cc07-946c-4119-bfb8-3f04130f52e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a439cc07-946c-4119-bfb8-3f04130f52e8-maas-api-openshift-ca\") pod \"a439cc07-946c-4119-bfb8-3f04130f52e8\" (UID: \"a439cc07-946c-4119-bfb8-3f04130f52e8\") "
Oct 30 05:19:41.194734 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:41.194740    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a439cc07-946c-4119-bfb8-3f04130f52e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a439cc07-946c-4119-bfb8-3f04130f52e8-maas-api-xks-ca\") pod \"a439cc07-946c-4119-bfb8-3f04130f52e8\" (UID: \"a439cc07-946c-4119-bfb8-3f04130f52e8\") "
Oct 30 05:19:41.194937 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:41.194762    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a439cc07-946c-4119-bfb8-3f04130f52e8-kube-api-access-jl8dr\" (UniqueName: \"kubernetes.io/projected/a439cc07-946c-4119-bfb8-3f04130f52e8-kube-api-access-jl8dr\") pod \"a439cc07-946c-4119-bfb8-3f04130f52e8\" (UID: \"a439cc07-946c-4119-bfb8-3f04130f52e8\") "
Oct 30 05:19:41.195064 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:41.195041    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a439cc07-946c-4119-bfb8-3f04130f52e8-maas-api-openshift-ca" pod "a439cc07-946c-4119-bfb8-3f04130f52e8" (UID: "a439cc07-946c-4119-bfb8-3f04130f52e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:19:41.195202 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:41.195172    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a439cc07-946c-4119-bfb8-3f04130f52e8-maas-api-openshift-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:19:41.196848 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:41.196816    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a439cc07-946c-4119-bfb8-3f04130f52e8-maas-api-xks-ca" pod "a439cc07-946c-4119-bfb8-3f04130f52e8" (UID: "a439cc07-946c-4119-bfb8-3f04130f52e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:19:41.197037 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:41.197017    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a439cc07-946c-4119-bfb8-3f04130f52e8-kube-api-access-jl8dr" pod "a439cc07-946c-4119-bfb8-3f04130f52e8" (UID: "a439cc07-946c-4119-bfb8-3f04130f52e8"). InnerVolumeSpecName "kube-api-access-jl8dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:19:41.295719 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:41.295681    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a439cc07-946c-4119-bfb8-3f04130f52e8-maas-api-xks-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:19:41.295719 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:41.295715    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl8dr\" (UniqueName: \"kubernetes.io/projected/a439cc07-946c-4119-bfb8-3f04130f52e8-kube-api-access-jl8dr\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:19:41.543917 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:41.543888    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4f5e14b-vp9nb"
Oct 30 05:19:41.544090 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:41.543918    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4f5e14b-vp9nb" event={"ID":"a439cc07-946c-4119-bfb8-3f04130f52e8","Type":"ContainerDied","Data":"6814c1d04a6e23bdaa6aa805e32eae3c396490f7829a43f7744641f46953c7dd"}
Oct 30 05:19:41.544090 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:41.543967    4874 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6814c1d04a6e23bdaa6aa805e32eae3c396490f7829a43f7744641f46953c7dd"
Oct 30 05:19:41.550658 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:41.550635    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:19:50.184889 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.184856    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"]
Oct 30 05:19:50.185413 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.185197    4874 cpu_manager.go:414] "containerMap: removing container" podUID="a439cc07-946c-4119-bfb8-3f04130f52e8" containerName="revoke-keys"
Oct 30 05:19:50.185413 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.185209    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a439cc07-946c-4119-bfb8-3f04130f52e8" containerName="revoke-keys"
Oct 30 05:19:50.185413 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.185324    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="a439cc07-946c-4119-bfb8-3f04130f52e8" containerName="revoke-keys"
Oct 30 05:19:50.188269 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.188227    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.204167 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.204146    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-6949ad-openshift-default-dockercfg-qbpg7\""
Oct 30 05:19:50.214463 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.214437    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e340979a-bec2-45fc-b742-604c80f1114f-istiod-ca-cert\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.214607 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.214484    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e340979a-bec2-45fc-b742-604c80f1114f-istio-token\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.214607 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.214506    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-workload-certs\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.214607 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.214579    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-workload-socket\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.214775 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.214643    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-credential-socket\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.214775 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.214740    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s25nw\" (UniqueName: \"kubernetes.io/projected/e340979a-bec2-45fc-b742-604c80f1114f-kube-api-access-s25nw\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.214890 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.214773    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e340979a-bec2-45fc-b742-604c80f1114f-istio-podinfo\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.214890 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.214804    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-istio-envoy\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.214890 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.214822    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-istio-data\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.217626 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.217602    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"]
Oct 30 05:19:50.315329 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.315292    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e340979a-bec2-45fc-b742-604c80f1114f-istiod-ca-cert\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.315484 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.315351    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e340979a-bec2-45fc-b742-604c80f1114f-istio-token\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.315484 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.315377    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-workload-certs\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.315484 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.315402    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-workload-socket\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.315484 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.315428    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-credential-socket\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.315725 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.315489    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s25nw\" (UniqueName: \"kubernetes.io/projected/e340979a-bec2-45fc-b742-604c80f1114f-kube-api-access-s25nw\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.315725 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.315523    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e340979a-bec2-45fc-b742-604c80f1114f-istio-podinfo\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.315725 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.315559    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-istio-envoy\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.315725 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.315589    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-istio-data\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.315958 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.315804    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-workload-certs\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.315958 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.315856    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-workload-socket\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.315958 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.315925    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-credential-socket\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.316086 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.315986    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-istio-data\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.316218 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.316097    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e340979a-bec2-45fc-b742-604c80f1114f-istiod-ca-cert\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.317905 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.317881    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-istio-envoy\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.318216 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.318198    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e340979a-bec2-45fc-b742-604c80f1114f-istio-podinfo\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.341981 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.341913    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e340979a-bec2-45fc-b742-604c80f1114f-istio-token\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.364807 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.364751    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s25nw\" (UniqueName: \"kubernetes.io/projected/e340979a-bec2-45fc-b742-604c80f1114f-kube-api-access-s25nw\") pod \"e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") " pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.500878 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.500851    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:50.814570 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.814534    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"]
Oct 30 05:19:50.815306 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:19:50.815280    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode340979a_bec2_45fc_b742_604c80f1114f.slice/crio-cd2637e224c10f69429f37610e822d0edd66c051ac7367ea129c889c11476ac6 WatchSource:0}: Error finding container cd2637e224c10f69429f37610e822d0edd66c051ac7367ea129c889c11476ac6: Status 404 returned error can't find the container with id cd2637e224c10f69429f37610e822d0edd66c051ac7367ea129c889c11476ac6
Oct 30 05:19:50.817393 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.817363    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 05:19:50.817466 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.817427    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 05:19:50.817466 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:50.817455    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 05:19:51.596649 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:51.596614    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q" event={"ID":"e340979a-bec2-45fc-b742-604c80f1114f","Type":"ContainerStarted","Data":"512abedc5b4f193f0390b2f5c9f731601f908f5b3a5ebdf2b78ce8727b946751"}
Oct 30 05:19:51.597027 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:51.596667    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q" event={"ID":"e340979a-bec2-45fc-b742-604c80f1114f","Type":"ContainerStarted","Data":"cd2637e224c10f69429f37610e822d0edd66c051ac7367ea129c889c11476ac6"}
Oct 30 05:19:51.653702 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:51.653658    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q" podStartSLOduration=1.65364502 podStartE2EDuration="1.65364502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:19:51.652682832 +0000 UTC m=+4911.065078417" watchObservedRunningTime="2026-10-30 05:19:51.65364502 +0000 UTC m=+4911.066040587"
Oct 30 05:19:52.501200 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:52.501162    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:52.506032 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:52.506007    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:52.601728 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:52.601705    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:52.602741 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:52.602722    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:19:55.608894 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:19:55.608853    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:19:55.787035 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:55.787004    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq"]
Oct 30 05:19:55.790866 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:55.790842    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq"
Oct 30 05:19:55.827688 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:55.827654    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq"]
Oct 30 05:19:55.832540 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:55.832515    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-6949ad-serving-cert\""
Oct 30 05:19:55.832659 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:55.832597    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gw5bk\""
Oct 30 05:19:55.832732 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:55.832710    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 05:19:55.867802 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:55.867714    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56vmm\" (UniqueName: \"kubernetes.io/projected/0de9acc3-a43d-4bde-9cc1-aa028192db23-kube-api-access-56vmm\") pod \"maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq\" (UID: \"0de9acc3-a43d-4bde-9cc1-aa028192db23\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq"
Oct 30 05:19:55.867802 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:55.867759    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0de9acc3-a43d-4bde-9cc1-aa028192db23-metrics-certs\") pod \"maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq\" (UID: \"0de9acc3-a43d-4bde-9cc1-aa028192db23\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq"
Oct 30 05:19:55.868055 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:55.867846    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0de9acc3-a43d-4bde-9cc1-aa028192db23-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq\" (UID: \"0de9acc3-a43d-4bde-9cc1-aa028192db23\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq"
Oct 30 05:19:55.868055 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:55.867921    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0de9acc3-a43d-4bde-9cc1-aa028192db23-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq\" (UID: \"0de9acc3-a43d-4bde-9cc1-aa028192db23\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq"
Oct 30 05:19:55.969110 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:55.969079    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56vmm\" (UniqueName: \"kubernetes.io/projected/0de9acc3-a43d-4bde-9cc1-aa028192db23-kube-api-access-56vmm\") pod \"maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq\" (UID: \"0de9acc3-a43d-4bde-9cc1-aa028192db23\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq"
Oct 30 05:19:55.969110 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:55.969114    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0de9acc3-a43d-4bde-9cc1-aa028192db23-metrics-certs\") pod \"maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq\" (UID: \"0de9acc3-a43d-4bde-9cc1-aa028192db23\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq"
Oct 30 05:19:55.969431 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:55.969138    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0de9acc3-a43d-4bde-9cc1-aa028192db23-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq\" (UID: \"0de9acc3-a43d-4bde-9cc1-aa028192db23\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq"
Oct 30 05:19:55.969431 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:55.969180    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0de9acc3-a43d-4bde-9cc1-aa028192db23-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq\" (UID: \"0de9acc3-a43d-4bde-9cc1-aa028192db23\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq"
Oct 30 05:19:55.970039 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:55.970009    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0de9acc3-a43d-4bde-9cc1-aa028192db23-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq\" (UID: \"0de9acc3-a43d-4bde-9cc1-aa028192db23\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq"
Oct 30 05:19:55.971659 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:55.971630    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0de9acc3-a43d-4bde-9cc1-aa028192db23-metrics-certs\") pod \"maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq\" (UID: \"0de9acc3-a43d-4bde-9cc1-aa028192db23\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq"
Oct 30 05:19:55.971748 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:55.971684    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0de9acc3-a43d-4bde-9cc1-aa028192db23-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq\" (UID: \"0de9acc3-a43d-4bde-9cc1-aa028192db23\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq"
Oct 30 05:19:56.020784 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.020750    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56vmm\" (UniqueName: \"kubernetes.io/projected/0de9acc3-a43d-4bde-9cc1-aa028192db23-kube-api-access-56vmm\") pod \"maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq\" (UID: \"0de9acc3-a43d-4bde-9cc1-aa028192db23\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq"
Oct 30 05:19:56.110423 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.110385    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq"
Oct 30 05:19:56.215755 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.215723    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7"]
Oct 30 05:19:56.219554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.219533    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7"
Oct 30 05:19:56.235122 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.235094    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk"]
Oct 30 05:19:56.238504 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.238484    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk"
Oct 30 05:19:56.245819 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.245801    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-6949ad\""
Oct 30 05:19:56.252262 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.251698    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-d5e9ff4a-dockercfg-j64rs\""
Oct 30 05:19:56.253173 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.253149    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7"]
Oct 30 05:19:56.300688 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.291444    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4d8h\" (UniqueName: \"kubernetes.io/projected/6141e561-11f5-42cb-90ec-680977c39d45-kube-api-access-d4d8h\") pod \"payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk\" (UID: \"6141e561-11f5-42cb-90ec-680977c39d45\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk"
Oct 30 05:19:56.300688 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.291519    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4175bf4d-8756-44e5-8e2d-9c02dbd985f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7\" (UID: \"4175bf4d-8756-44e5-8e2d-9c02dbd985f0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7"
Oct 30 05:19:56.300688 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.291571    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6141e561-11f5-42cb-90ec-680977c39d45-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk\" (UID: \"6141e561-11f5-42cb-90ec-680977c39d45\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk"
Oct 30 05:19:56.300688 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.291615    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj58p\" (UniqueName: \"kubernetes.io/projected/4175bf4d-8756-44e5-8e2d-9c02dbd985f0-kube-api-access-bj58p\") pod \"payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7\" (UID: \"4175bf4d-8756-44e5-8e2d-9c02dbd985f0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7"
Oct 30 05:19:56.300688 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.297325    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk"]
Oct 30 05:19:56.392227 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.392154    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4d8h\" (UniqueName: \"kubernetes.io/projected/6141e561-11f5-42cb-90ec-680977c39d45-kube-api-access-d4d8h\") pod \"payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk\" (UID: \"6141e561-11f5-42cb-90ec-680977c39d45\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk"
Oct 30 05:19:56.392227 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.392204    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4175bf4d-8756-44e5-8e2d-9c02dbd985f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7\" (UID: \"4175bf4d-8756-44e5-8e2d-9c02dbd985f0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7"
Oct 30 05:19:56.392227 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.392226    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6141e561-11f5-42cb-90ec-680977c39d45-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk\" (UID: \"6141e561-11f5-42cb-90ec-680977c39d45\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk"
Oct 30 05:19:56.392505 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.392361    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj58p\" (UniqueName: \"kubernetes.io/projected/4175bf4d-8756-44e5-8e2d-9c02dbd985f0-kube-api-access-bj58p\") pod \"payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7\" (UID: \"4175bf4d-8756-44e5-8e2d-9c02dbd985f0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7"
Oct 30 05:19:56.392969 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.392948    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4175bf4d-8756-44e5-8e2d-9c02dbd985f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7\" (UID: \"4175bf4d-8756-44e5-8e2d-9c02dbd985f0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7"
Oct 30 05:19:56.393060 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.392948    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6141e561-11f5-42cb-90ec-680977c39d45-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk\" (UID: \"6141e561-11f5-42cb-90ec-680977c39d45\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk"
Oct 30 05:19:56.415040 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.415011    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4d8h\" (UniqueName: \"kubernetes.io/projected/6141e561-11f5-42cb-90ec-680977c39d45-kube-api-access-d4d8h\") pod \"payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk\" (UID: \"6141e561-11f5-42cb-90ec-680977c39d45\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk"
Oct 30 05:19:56.418965 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.418945    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj58p\" (UniqueName: \"kubernetes.io/projected/4175bf4d-8756-44e5-8e2d-9c02dbd985f0-kube-api-access-bj58p\") pod \"payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7\" (UID: \"4175bf4d-8756-44e5-8e2d-9c02dbd985f0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7"
Oct 30 05:19:56.421269 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.421231    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq"]
Oct 30 05:19:56.422699 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:19:56.422672    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de9acc3_a43d_4bde_9cc1_aa028192db23.slice/crio-cc22251948afb44ec22c9b74d3dcdd76b047792f87765c367f7ec51257e7f6de WatchSource:0}: Error finding container cc22251948afb44ec22c9b74d3dcdd76b047792f87765c367f7ec51257e7f6de: Status 404 returned error can't find the container with id cc22251948afb44ec22c9b74d3dcdd76b047792f87765c367f7ec51257e7f6de
Oct 30 05:19:56.532742 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.532711    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7"
Oct 30 05:19:56.551092 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.551069    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk"
Oct 30 05:19:56.623685 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.623648    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq" event={"ID":"0de9acc3-a43d-4bde-9cc1-aa028192db23","Type":"ContainerStarted","Data":"cc22251948afb44ec22c9b74d3dcdd76b047792f87765c367f7ec51257e7f6de"}
Oct 30 05:19:56.833121 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.832975    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7"]
Oct 30 05:19:56.835275 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:19:56.835228    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4175bf4d_8756_44e5_8e2d_9c02dbd985f0.slice/crio-5715a9d11c69fda61db66e31780a55352ba2e0cd9d128996f729fc8041a7bf33 WatchSource:0}: Error finding container 5715a9d11c69fda61db66e31780a55352ba2e0cd9d128996f729fc8041a7bf33: Status 404 returned error can't find the container with id 5715a9d11c69fda61db66e31780a55352ba2e0cd9d128996f729fc8041a7bf33
Oct 30 05:19:56.862173 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:56.862149    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk"]
Oct 30 05:19:56.862301 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:19:56.862199    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6141e561_11f5_42cb_90ec_680977c39d45.slice/crio-7846e5070704bdb59708bc02c7b08d1dfcbb049a79a1306ba862041c14f57224 WatchSource:0}: Error finding container 7846e5070704bdb59708bc02c7b08d1dfcbb049a79a1306ba862041c14f57224: Status 404 returned error can't find the container with id 7846e5070704bdb59708bc02c7b08d1dfcbb049a79a1306ba862041c14f57224
Oct 30 05:19:57.630042 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:57.630009    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq" event={"ID":"0de9acc3-a43d-4bde-9cc1-aa028192db23","Type":"ContainerStarted","Data":"1d86684e698c6eb30988a11b8a56cbfdc1f905f77622900fc4235247a6585e85"}
Oct 30 05:19:57.630566 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:57.630106    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq"
Oct 30 05:19:57.631698 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:57.631673    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7" event={"ID":"4175bf4d-8756-44e5-8e2d-9c02dbd985f0","Type":"ContainerStarted","Data":"63ef0f839a911129ca9ff817091791bbb375ae58114ee942a8aada35147b51e1"}
Oct 30 05:19:57.631842 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:57.631712    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7" event={"ID":"4175bf4d-8756-44e5-8e2d-9c02dbd985f0","Type":"ContainerStarted","Data":"5715a9d11c69fda61db66e31780a55352ba2e0cd9d128996f729fc8041a7bf33"}
Oct 30 05:19:57.631842 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:57.631738    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7"
Oct 30 05:19:57.633146 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:57.633127    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk" event={"ID":"6141e561-11f5-42cb-90ec-680977c39d45","Type":"ContainerStarted","Data":"809b687734b4127b9781ef8015227ebaf27f474ec4cbc8dd1d40501a8888ae5e"}
Oct 30 05:19:57.633256 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:57.633160    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk" event={"ID":"6141e561-11f5-42cb-90ec-680977c39d45","Type":"ContainerStarted","Data":"7846e5070704bdb59708bc02c7b08d1dfcbb049a79a1306ba862041c14f57224"}
Oct 30 05:19:57.633256 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:57.633181    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk"
Oct 30 05:19:57.673082 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:57.673043    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq" podStartSLOduration=2.064090579 podStartE2EDuration="2.67303025s" totalImagesPullingTime="608.939671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:19:57.671844329 +0000 UTC m=+4917.084239901" watchObservedRunningTime="2026-10-30 05:19:57.67303025 +0000 UTC m=+4917.085425822"
Oct 30 05:19:57.711755 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:57.711710    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk" podStartSLOduration=1.711695763 podStartE2EDuration="1.711695763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:19:57.710069467 +0000 UTC m=+4917.122465040" watchObservedRunningTime="2026-10-30 05:19:57.711695763 +0000 UTC m=+4917.124091338"
Oct 30 05:19:57.749702 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:57.749655    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7" podStartSLOduration=1.74964169 podStartE2EDuration="1.74964169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:19:57.748555328 +0000 UTC m=+4917.160950902" watchObservedRunningTime="2026-10-30 05:19:57.74964169 +0000 UTC m=+4917.162037261"
Oct 30 05:19:59.865669 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:19:59.865626    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:20:03.644045 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:03.644016    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq"
Oct 30 05:20:08.640369 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:08.640334    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk"
Oct 30 05:20:08.640834 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:08.640585    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7"
Oct 30 05:20:08.645367 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:08.645343    4874 patch_prober.go:29] interesting pod/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:20:08.645498 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:08.645395    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q" podUID="e340979a-bec2-45fc-b742-604c80f1114f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:20:09.726762 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:09.726735    4874 generic.go:356] "Generic (PLEG): container finished" podID="e340979a-bec2-45fc-b742-604c80f1114f" containerID="512abedc5b4f193f0390b2f5c9f731601f908f5b3a5ebdf2b78ce8727b946751" exitCode=0
Oct 30 05:20:09.727168 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:09.726811    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q" event={"ID":"e340979a-bec2-45fc-b742-604c80f1114f","Type":"ContainerDied","Data":"512abedc5b4f193f0390b2f5c9f731601f908f5b3a5ebdf2b78ce8727b946751"}
Oct 30 05:20:09.727168 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:09.727128    4874 scope.go:122] "RemoveContainer" containerID="512abedc5b4f193f0390b2f5c9f731601f908f5b3a5ebdf2b78ce8727b946751"
Oct 30 05:20:09.728327 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:09.728300    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 05:20:09.728393 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:09.728367    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 05:20:09.728439 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:09.728402    4874 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 05:20:10.732449 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:10.732410    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q" event={"ID":"e340979a-bec2-45fc-b742-604c80f1114f","Type":"ContainerStarted","Data":"71f14a9e1137320803fd9e6aa8fade67c07a6704798463d1aafe070a6815863c"}
Oct 30 05:20:11.501581 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.501547    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:20:11.501765 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.501742    4874 patch_prober.go:29] interesting pod/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 30 05:20:11.501813 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.501790    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q" podUID="e340979a-bec2-45fc-b742-604c80f1114f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 30 05:20:11.563008 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.562978    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"]
Oct 30 05:20:11.582511 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.582484    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:20:11.591606 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.591574    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"]
Oct 30 05:20:11.617697 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.617637    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-6949ad\"/\"openshift-service-ca.crt\""
Oct 30 05:20:11.617697 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.617649    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-6949ad\"/\"kube-root-ca.crt\""
Oct 30 05:20:11.617697 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.617695    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-6949ad\"/\"simulator-single-main-6949ad-kserve-self-signed-certs\""
Oct 30 05:20:11.618026 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.618011    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-6949ad\"/\"default-dockercfg-2kvrw\""
Oct 30 05:20:11.663711 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.663686    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5nwm\" (UniqueName: \"kubernetes.io/projected/b259f029-bdc1-4880-aeb7-cc3a47c8216b-kube-api-access-r5nwm\") pod \"simulator-single-main-6949ad-kserve-785c48c897-42tk8\" (UID: \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\") " pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:20:11.663843 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.663761    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-home\") pod \"simulator-single-main-6949ad-kserve-785c48c897-42tk8\" (UID: \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\") " pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:20:11.663843 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.663832    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b259f029-bdc1-4880-aeb7-cc3a47c8216b-tls-certs\") pod \"simulator-single-main-6949ad-kserve-785c48c897-42tk8\" (UID: \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\") " pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:20:11.663959 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.663857    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-tmp-dir\") pod \"simulator-single-main-6949ad-kserve-785c48c897-42tk8\" (UID: \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\") " pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:20:11.663959 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.663901    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-model-cache\") pod \"simulator-single-main-6949ad-kserve-785c48c897-42tk8\" (UID: \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\") " pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:20:11.664073 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.663966    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-dshm\") pod \"simulator-single-main-6949ad-kserve-785c48c897-42tk8\" (UID: \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\") " pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:20:11.766225 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.766138    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-home\") pod \"simulator-single-main-6949ad-kserve-785c48c897-42tk8\" (UID: \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\") " pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:20:11.766828 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.766803    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b259f029-bdc1-4880-aeb7-cc3a47c8216b-tls-certs\") pod \"simulator-single-main-6949ad-kserve-785c48c897-42tk8\" (UID: \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\") " pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:20:11.766968 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.766952    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-tmp-dir\") pod \"simulator-single-main-6949ad-kserve-785c48c897-42tk8\" (UID: \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\") " pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:20:11.767144 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.767125    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-model-cache\") pod \"simulator-single-main-6949ad-kserve-785c48c897-42tk8\" (UID: \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\") " pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:20:11.769052 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.769031    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-dshm\") pod \"simulator-single-main-6949ad-kserve-785c48c897-42tk8\" (UID: \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\") " pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:20:11.769272 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.767912    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-home\") pod \"simulator-single-main-6949ad-kserve-785c48c897-42tk8\" (UID: \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\") " pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:20:11.769468 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.767700    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-tmp-dir\") pod \"simulator-single-main-6949ad-kserve-785c48c897-42tk8\" (UID: \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\") " pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:20:11.770398 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.769889    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5nwm\" (UniqueName: \"kubernetes.io/projected/b259f029-bdc1-4880-aeb7-cc3a47c8216b-kube-api-access-r5nwm\") pod \"simulator-single-main-6949ad-kserve-785c48c897-42tk8\" (UID: \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\") " pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:20:11.771015 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.770977    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-model-cache\") pod \"simulator-single-main-6949ad-kserve-785c48c897-42tk8\" (UID: \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\") " pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:20:11.788789 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.788766    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b259f029-bdc1-4880-aeb7-cc3a47c8216b-tls-certs\") pod \"simulator-single-main-6949ad-kserve-785c48c897-42tk8\" (UID: \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\") " pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:20:11.793258 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.791051    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-dshm\") pod \"simulator-single-main-6949ad-kserve-785c48c897-42tk8\" (UID: \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\") " pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:20:11.797907 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.797875    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5nwm\" (UniqueName: \"kubernetes.io/projected/b259f029-bdc1-4880-aeb7-cc3a47c8216b-kube-api-access-r5nwm\") pod \"simulator-single-main-6949ad-kserve-785c48c897-42tk8\" (UID: \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\") " pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:20:11.906169 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:11.906132    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:20:12.219656 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:12.219629    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"]
Oct 30 05:20:12.220458 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:20:12.220430    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb259f029_bdc1_4880_aeb7_cc3a47c8216b.slice/crio-fb3ef2c3b00418d2bcc4700741b4082163afe16b4ec648a258445938ad9f58aa WatchSource:0}: Error finding container fb3ef2c3b00418d2bcc4700741b4082163afe16b4ec648a258445938ad9f58aa: Status 404 returned error can't find the container with id fb3ef2c3b00418d2bcc4700741b4082163afe16b4ec648a258445938ad9f58aa
Oct 30 05:20:12.502049 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:12.502019    4874 patch_prober.go:29] interesting pod/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 30 05:20:12.502218 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:12.502069    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q" podUID="e340979a-bec2-45fc-b742-604c80f1114f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 30 05:20:12.742966 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:12.742926    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8" event={"ID":"b259f029-bdc1-4880-aeb7-cc3a47c8216b","Type":"ContainerStarted","Data":"44c4bc48e201566684e470ea8d7e989b89805f88ee12bdb633db10b3dd0da531"}
Oct 30 05:20:12.742966 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:12.742973    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8" event={"ID":"b259f029-bdc1-4880-aeb7-cc3a47c8216b","Type":"ContainerStarted","Data":"fb3ef2c3b00418d2bcc4700741b4082163afe16b4ec648a258445938ad9f58aa"}
Oct 30 05:20:12.791326 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:12.791201    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8" podStartSLOduration=1.791187592 podStartE2EDuration="1.791187592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:20:12.790590563 +0000 UTC m=+4932.202986139" watchObservedRunningTime="2026-10-30 05:20:12.791187592 +0000 UTC m=+4932.203583161"
Oct 30 05:20:12.911262 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:12.907896    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:20:13.501288 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:13.501249    4874 patch_prober.go:29] interesting pod/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 30 05:20:13.501498 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:13.501321    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q" podUID="e340979a-bec2-45fc-b742-604c80f1114f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 30 05:20:14.036350 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:14.036315    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:20:14.504676 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:14.504636    4874 patch_prober.go:29] interesting pod/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 30 05:20:14.505010 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:14.504978    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q" podUID="e340979a-bec2-45fc-b742-604c80f1114f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 30 05:20:15.502003 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:15.501958    4874 patch_prober.go:29] interesting pod/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 30 05:20:15.502513 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:15.502025    4874 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q" podUID="e340979a-bec2-45fc-b742-604c80f1114f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 30 05:20:16.504702 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:16.504670    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:20:16.505098 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:16.504946    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:20:16.505696 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:16.505680    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:20:21.906646 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:21.906613    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:20:21.919179 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:21.919148    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:20:22.818345 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:22.818315    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:20:25.608768 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:20:25.608725    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:20:31.444145 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:31.444107    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:20:55.608681 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:20:55.608645    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:20:56.113630 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:20:56.113599    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:21:00.314844 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:00.314810    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"]
Oct 30 05:21:02.937895 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:02.937867    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk"]
Oct 30 05:21:02.979317 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:02.979289    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7"]
Oct 30 05:21:05.451648 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:05.451613    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq"]
Oct 30 05:21:05.452162 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:05.451900    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq" podUID="0de9acc3-a43d-4bde-9cc1-aa028192db23" containerName="maas-api" containerID="cri-o://1d86684e698c6eb30988a11b8a56cbfdc1f905f77622900fc4235247a6585e85" gracePeriod=30
Oct 30 05:21:06.794007 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:06.793985    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq"
Oct 30 05:21:06.854034 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:06.853965    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de9acc3-a43d-4bde-9cc1-aa028192db23-kube-api-access-56vmm\" (UniqueName: \"kubernetes.io/projected/0de9acc3-a43d-4bde-9cc1-aa028192db23-kube-api-access-56vmm\") pod \"0de9acc3-a43d-4bde-9cc1-aa028192db23\" (UID: \"0de9acc3-a43d-4bde-9cc1-aa028192db23\") "
Oct 30 05:21:06.854194 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:06.854086    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0de9acc3-a43d-4bde-9cc1-aa028192db23-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0de9acc3-a43d-4bde-9cc1-aa028192db23-maas-api-tls\") pod \"0de9acc3-a43d-4bde-9cc1-aa028192db23\" (UID: \"0de9acc3-a43d-4bde-9cc1-aa028192db23\") "
Oct 30 05:21:06.854194 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:06.854128    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0de9acc3-a43d-4bde-9cc1-aa028192db23-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0de9acc3-a43d-4bde-9cc1-aa028192db23-metrics-certs\") pod \"0de9acc3-a43d-4bde-9cc1-aa028192db23\" (UID: \"0de9acc3-a43d-4bde-9cc1-aa028192db23\") "
Oct 30 05:21:06.854194 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:06.854157    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0de9acc3-a43d-4bde-9cc1-aa028192db23-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0de9acc3-a43d-4bde-9cc1-aa028192db23-openshift-service-ca\") pod \"0de9acc3-a43d-4bde-9cc1-aa028192db23\" (UID: \"0de9acc3-a43d-4bde-9cc1-aa028192db23\") "
Oct 30 05:21:06.854638 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:06.854605    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0de9acc3-a43d-4bde-9cc1-aa028192db23-openshift-service-ca" pod "0de9acc3-a43d-4bde-9cc1-aa028192db23" (UID: "0de9acc3-a43d-4bde-9cc1-aa028192db23"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:21:06.856267 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:06.856217    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0de9acc3-a43d-4bde-9cc1-aa028192db23-metrics-certs" pod "0de9acc3-a43d-4bde-9cc1-aa028192db23" (UID: "0de9acc3-a43d-4bde-9cc1-aa028192db23"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:21:06.856385 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:06.856293    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0de9acc3-a43d-4bde-9cc1-aa028192db23-maas-api-tls" pod "0de9acc3-a43d-4bde-9cc1-aa028192db23" (UID: "0de9acc3-a43d-4bde-9cc1-aa028192db23"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:21:06.856385 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:06.856322    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de9acc3-a43d-4bde-9cc1-aa028192db23-kube-api-access-56vmm" pod "0de9acc3-a43d-4bde-9cc1-aa028192db23" (UID: "0de9acc3-a43d-4bde-9cc1-aa028192db23"). InnerVolumeSpecName "kube-api-access-56vmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:21:06.958667 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:06.958630    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0de9acc3-a43d-4bde-9cc1-aa028192db23-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:21:06.958667 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:06.958669    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0de9acc3-a43d-4bde-9cc1-aa028192db23-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:21:06.958868 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:06.958684    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0de9acc3-a43d-4bde-9cc1-aa028192db23-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:21:06.958868 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:06.958698    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56vmm\" (UniqueName: \"kubernetes.io/projected/0de9acc3-a43d-4bde-9cc1-aa028192db23-kube-api-access-56vmm\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:21:07.012323 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:07.012291    4874 generic.go:356] "Generic (PLEG): container finished" podID="0de9acc3-a43d-4bde-9cc1-aa028192db23" containerID="1d86684e698c6eb30988a11b8a56cbfdc1f905f77622900fc4235247a6585e85" exitCode=0
Oct 30 05:21:07.012473 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:07.012335    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq"
Oct 30 05:21:07.012473 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:07.012371    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq" event={"ID":"0de9acc3-a43d-4bde-9cc1-aa028192db23","Type":"ContainerDied","Data":"1d86684e698c6eb30988a11b8a56cbfdc1f905f77622900fc4235247a6585e85"}
Oct 30 05:21:07.012473 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:07.012432    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq" event={"ID":"0de9acc3-a43d-4bde-9cc1-aa028192db23","Type":"ContainerDied","Data":"cc22251948afb44ec22c9b74d3dcdd76b047792f87765c367f7ec51257e7f6de"}
Oct 30 05:21:07.012473 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:07.012455    4874 scope.go:122] "RemoveContainer" containerID="1d86684e698c6eb30988a11b8a56cbfdc1f905f77622900fc4235247a6585e85"
Oct 30 05:21:07.021494 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:07.021475    4874 scope.go:122] "RemoveContainer" containerID="1d86684e698c6eb30988a11b8a56cbfdc1f905f77622900fc4235247a6585e85"
Oct 30 05:21:07.021756 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:21:07.021735    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d86684e698c6eb30988a11b8a56cbfdc1f905f77622900fc4235247a6585e85\": container with ID starting with 1d86684e698c6eb30988a11b8a56cbfdc1f905f77622900fc4235247a6585e85 not found: ID does not exist" containerID="1d86684e698c6eb30988a11b8a56cbfdc1f905f77622900fc4235247a6585e85"
Oct 30 05:21:07.021821 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:07.021771    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d86684e698c6eb30988a11b8a56cbfdc1f905f77622900fc4235247a6585e85"} err="failed to get container status \"1d86684e698c6eb30988a11b8a56cbfdc1f905f77622900fc4235247a6585e85\": rpc error: code = NotFound desc = could not find container \"1d86684e698c6eb30988a11b8a56cbfdc1f905f77622900fc4235247a6585e85\": container with ID starting with 1d86684e698c6eb30988a11b8a56cbfdc1f905f77622900fc4235247a6585e85 not found: ID does not exist"
Oct 30 05:21:07.069833 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:07.069809    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq"]
Oct 30 05:21:07.072493 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:07.072467    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6949ad-6999f4cccf-49csq"]
Oct 30 05:21:07.924373 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:07.924324    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:21:07.938643 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:07.938597    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk" podUID="6141e561-11f5-42cb-90ec-680977c39d45" containerName="payload-processing" containerID="cri-o://809b687734b4127b9781ef8015227ebaf27f474ec4cbc8dd1d40501a8888ae5e" gracePeriod=25
Oct 30 05:21:07.980570 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:07.980536    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7" podUID="4175bf4d-8756-44e5-8e2d-9c02dbd985f0" containerName="payload-pre-processing" containerID="cri-o://63ef0f839a911129ca9ff817091791bbb375ae58114ee942a8aada35147b51e1" gracePeriod=25
Oct 30 05:21:08.638997 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:08.638947    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7" podUID="4175bf4d-8756-44e5-8e2d-9c02dbd985f0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:21:08.639181 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:08.638947    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk" podUID="6141e561-11f5-42cb-90ec-680977c39d45" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:21:08.727777 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:08.727748    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0de9acc3-a43d-4bde-9cc1-aa028192db23" path="/var/lib/kubelet/pods/0de9acc3-a43d-4bde-9cc1-aa028192db23/volumes"
Oct 30 05:21:11.803306 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:11.803274    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"]
Oct 30 05:21:11.803790 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:11.803457    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q" podUID="e340979a-bec2-45fc-b742-604c80f1114f" containerName="istio-proxy" containerID="cri-o://71f14a9e1137320803fd9e6aa8fade67c07a6704798463d1aafe070a6815863c" gracePeriod=30
Oct 30 05:21:15.334228 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:15.334130    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8" podUID="b259f029-bdc1-4880-aeb7-cc3a47c8216b" containerName="main" containerID="cri-o://44c4bc48e201566684e470ea8d7e989b89805f88ee12bdb633db10b3dd0da531" gracePeriod=45
Oct 30 05:21:15.580644 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:15.580619    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:21:15.656587 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:15.656510    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-model-cache\") pod \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\" (UID: \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\") "
Oct 30 05:21:15.656587 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:15.656560    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b259f029-bdc1-4880-aeb7-cc3a47c8216b-tls-certs\" (UniqueName: \"kubernetes.io/secret/b259f029-bdc1-4880-aeb7-cc3a47c8216b-tls-certs\") pod \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\" (UID: \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\") "
Oct 30 05:21:15.656784 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:15.656609    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-home\" (UniqueName: \"kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-home\") pod \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\" (UID: \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\") "
Oct 30 05:21:15.656784 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:15.656648    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-tmp-dir\") pod \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\" (UID: \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\") "
Oct 30 05:21:15.656784 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:15.656669    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-dshm\") pod \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\" (UID: \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\") "
Oct 30 05:21:15.656784 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:15.656754    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b259f029-bdc1-4880-aeb7-cc3a47c8216b-kube-api-access-r5nwm\" (UniqueName: \"kubernetes.io/projected/b259f029-bdc1-4880-aeb7-cc3a47c8216b-kube-api-access-r5nwm\") pod \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\" (UID: \"b259f029-bdc1-4880-aeb7-cc3a47c8216b\") "
Oct 30 05:21:15.657021 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:15.656808    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-model-cache" pod "b259f029-bdc1-4880-aeb7-cc3a47c8216b" (UID: "b259f029-bdc1-4880-aeb7-cc3a47c8216b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:21:15.657021 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:15.656904    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-tmp-dir" pod "b259f029-bdc1-4880-aeb7-cc3a47c8216b" (UID: "b259f029-bdc1-4880-aeb7-cc3a47c8216b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:21:15.657021 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:15.656921    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-home" pod "b259f029-bdc1-4880-aeb7-cc3a47c8216b" (UID: "b259f029-bdc1-4880-aeb7-cc3a47c8216b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:21:15.657164 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:15.657096    4874 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-home\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:21:15.657164 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:15.657116    4874 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-tmp-dir\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:21:15.657164 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:15.657130    4874 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-model-cache\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:21:15.658831 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:15.658809    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b259f029-bdc1-4880-aeb7-cc3a47c8216b-tls-certs" pod "b259f029-bdc1-4880-aeb7-cc3a47c8216b" (UID: "b259f029-bdc1-4880-aeb7-cc3a47c8216b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:21:15.658969 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:15.658883    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b259f029-bdc1-4880-aeb7-cc3a47c8216b-kube-api-access-r5nwm" pod "b259f029-bdc1-4880-aeb7-cc3a47c8216b" (UID: "b259f029-bdc1-4880-aeb7-cc3a47c8216b"). InnerVolumeSpecName "kube-api-access-r5nwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:21:15.658969 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:15.658932    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-dshm" pod "b259f029-bdc1-4880-aeb7-cc3a47c8216b" (UID: "b259f029-bdc1-4880-aeb7-cc3a47c8216b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:21:15.757623 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:15.757594    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5nwm\" (UniqueName: \"kubernetes.io/projected/b259f029-bdc1-4880-aeb7-cc3a47c8216b-kube-api-access-r5nwm\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:21:15.757623 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:15.757621    4874 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b259f029-bdc1-4880-aeb7-cc3a47c8216b-tls-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:21:15.757623 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:15.757630    4874 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b259f029-bdc1-4880-aeb7-cc3a47c8216b-dshm\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:21:16.065822 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:16.065787    4874 generic.go:356] "Generic (PLEG): container finished" podID="b259f029-bdc1-4880-aeb7-cc3a47c8216b" containerID="44c4bc48e201566684e470ea8d7e989b89805f88ee12bdb633db10b3dd0da531" exitCode=0
Oct 30 05:21:16.066025 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:16.065835    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"
Oct 30 05:21:16.066025 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:16.065848    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8" event={"ID":"b259f029-bdc1-4880-aeb7-cc3a47c8216b","Type":"ContainerDied","Data":"44c4bc48e201566684e470ea8d7e989b89805f88ee12bdb633db10b3dd0da531"}
Oct 30 05:21:16.066025 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:16.065879    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8" event={"ID":"b259f029-bdc1-4880-aeb7-cc3a47c8216b","Type":"ContainerDied","Data":"fb3ef2c3b00418d2bcc4700741b4082163afe16b4ec648a258445938ad9f58aa"}
Oct 30 05:21:16.066025 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:16.065895    4874 scope.go:122] "RemoveContainer" containerID="44c4bc48e201566684e470ea8d7e989b89805f88ee12bdb633db10b3dd0da531"
Oct 30 05:21:16.075598 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:16.075576    4874 scope.go:122] "RemoveContainer" containerID="44c4bc48e201566684e470ea8d7e989b89805f88ee12bdb633db10b3dd0da531"
Oct 30 05:21:16.075887 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:21:16.075868    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c4bc48e201566684e470ea8d7e989b89805f88ee12bdb633db10b3dd0da531\": container with ID starting with 44c4bc48e201566684e470ea8d7e989b89805f88ee12bdb633db10b3dd0da531 not found: ID does not exist" containerID="44c4bc48e201566684e470ea8d7e989b89805f88ee12bdb633db10b3dd0da531"
Oct 30 05:21:16.075950 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:16.075894    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c4bc48e201566684e470ea8d7e989b89805f88ee12bdb633db10b3dd0da531"} err="failed to get container status \"44c4bc48e201566684e470ea8d7e989b89805f88ee12bdb633db10b3dd0da531\": rpc error: code = NotFound desc = could not find container \"44c4bc48e201566684e470ea8d7e989b89805f88ee12bdb633db10b3dd0da531\": container with ID starting with 44c4bc48e201566684e470ea8d7e989b89805f88ee12bdb633db10b3dd0da531 not found: ID does not exist"
Oct 30 05:21:16.120041 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:16.120017    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"]
Oct 30 05:21:16.131222 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:16.131203    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-6949ad/simulator-single-main-6949ad-kserve-785c48c897-42tk8"]
Oct 30 05:21:16.507492 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:16.507463    4874 patch_prober.go:29] interesting pod/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 05:21:16.507873 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:16.507520    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q" podUID="e340979a-bec2-45fc-b742-604c80f1114f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 05:21:16.728037 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:16.728004    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b259f029-bdc1-4880-aeb7-cc3a47c8216b" path="/var/lib/kubelet/pods/b259f029-bdc1-4880-aeb7-cc3a47c8216b/volumes"
Oct 30 05:21:17.053106 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.053081    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:21:17.074011 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.073947    4874 generic.go:356] "Generic (PLEG): container finished" podID="e340979a-bec2-45fc-b742-604c80f1114f" containerID="71f14a9e1137320803fd9e6aa8fade67c07a6704798463d1aafe070a6815863c" exitCode=0
Oct 30 05:21:17.074011 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.073991    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"
Oct 30 05:21:17.074175 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.074023    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q" event={"ID":"e340979a-bec2-45fc-b742-604c80f1114f","Type":"ContainerDied","Data":"71f14a9e1137320803fd9e6aa8fade67c07a6704798463d1aafe070a6815863c"}
Oct 30 05:21:17.074175 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.074077    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q" event={"ID":"e340979a-bec2-45fc-b742-604c80f1114f","Type":"ContainerDied","Data":"cd2637e224c10f69429f37610e822d0edd66c051ac7367ea129c889c11476ac6"}
Oct 30 05:21:17.074175 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.074102    4874 scope.go:122] "RemoveContainer" containerID="71f14a9e1137320803fd9e6aa8fade67c07a6704798463d1aafe070a6815863c"
Oct 30 05:21:17.082537 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.082520    4874 scope.go:122] "RemoveContainer" containerID="512abedc5b4f193f0390b2f5c9f731601f908f5b3a5ebdf2b78ce8727b946751"
Oct 30 05:21:17.090977 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.090955    4874 scope.go:122] "RemoveContainer" containerID="71f14a9e1137320803fd9e6aa8fade67c07a6704798463d1aafe070a6815863c"
Oct 30 05:21:17.091223 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:21:17.091206    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71f14a9e1137320803fd9e6aa8fade67c07a6704798463d1aafe070a6815863c\": container with ID starting with 71f14a9e1137320803fd9e6aa8fade67c07a6704798463d1aafe070a6815863c not found: ID does not exist" containerID="71f14a9e1137320803fd9e6aa8fade67c07a6704798463d1aafe070a6815863c"
Oct 30 05:21:17.091310 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.091233    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f14a9e1137320803fd9e6aa8fade67c07a6704798463d1aafe070a6815863c"} err="failed to get container status \"71f14a9e1137320803fd9e6aa8fade67c07a6704798463d1aafe070a6815863c\": rpc error: code = NotFound desc = could not find container \"71f14a9e1137320803fd9e6aa8fade67c07a6704798463d1aafe070a6815863c\": container with ID starting with 71f14a9e1137320803fd9e6aa8fade67c07a6704798463d1aafe070a6815863c not found: ID does not exist"
Oct 30 05:21:17.091310 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.091267    4874 scope.go:122] "RemoveContainer" containerID="512abedc5b4f193f0390b2f5c9f731601f908f5b3a5ebdf2b78ce8727b946751"
Oct 30 05:21:17.091500 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:21:17.091473    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"512abedc5b4f193f0390b2f5c9f731601f908f5b3a5ebdf2b78ce8727b946751\": container with ID starting with 512abedc5b4f193f0390b2f5c9f731601f908f5b3a5ebdf2b78ce8727b946751 not found: ID does not exist" containerID="512abedc5b4f193f0390b2f5c9f731601f908f5b3a5ebdf2b78ce8727b946751"
Oct 30 05:21:17.091552 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.091510    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"512abedc5b4f193f0390b2f5c9f731601f908f5b3a5ebdf2b78ce8727b946751"} err="failed to get container status \"512abedc5b4f193f0390b2f5c9f731601f908f5b3a5ebdf2b78ce8727b946751\": rpc error: code = NotFound desc = could not find container \"512abedc5b4f193f0390b2f5c9f731601f908f5b3a5ebdf2b78ce8727b946751\": container with ID starting with 512abedc5b4f193f0390b2f5c9f731601f908f5b3a5ebdf2b78ce8727b946751 not found: ID does not exist"
Oct 30 05:21:17.169851 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.169821    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e340979a-bec2-45fc-b742-604c80f1114f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e340979a-bec2-45fc-b742-604c80f1114f-istiod-ca-cert\") pod \"e340979a-bec2-45fc-b742-604c80f1114f\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") "
Oct 30 05:21:17.169993 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.169859    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e340979a-bec2-45fc-b742-604c80f1114f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e340979a-bec2-45fc-b742-604c80f1114f-istio-podinfo\") pod \"e340979a-bec2-45fc-b742-604c80f1114f\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") "
Oct 30 05:21:17.169993 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.169904    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-credential-socket\") pod \"e340979a-bec2-45fc-b742-604c80f1114f\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") "
Oct 30 05:21:17.169993 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.169943    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-istio-envoy\") pod \"e340979a-bec2-45fc-b742-604c80f1114f\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") "
Oct 30 05:21:17.170171 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.170022    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-workload-certs\") pod \"e340979a-bec2-45fc-b742-604c80f1114f\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") "
Oct 30 05:21:17.170171 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.170095    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e340979a-bec2-45fc-b742-604c80f1114f-kube-api-access-s25nw\" (UniqueName: \"kubernetes.io/projected/e340979a-bec2-45fc-b742-604c80f1114f-kube-api-access-s25nw\") pod \"e340979a-bec2-45fc-b742-604c80f1114f\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") "
Oct 30 05:21:17.170343 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.170223    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e340979a-bec2-45fc-b742-604c80f1114f-istio-token\" (UniqueName: \"kubernetes.io/projected/e340979a-bec2-45fc-b742-604c80f1114f-istio-token\") pod \"e340979a-bec2-45fc-b742-604c80f1114f\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") "
Oct 30 05:21:17.170343 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.170298    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-workload-socket\") pod \"e340979a-bec2-45fc-b742-604c80f1114f\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") "
Oct 30 05:21:17.170343 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.170332    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-istio-data\") pod \"e340979a-bec2-45fc-b742-604c80f1114f\" (UID: \"e340979a-bec2-45fc-b742-604c80f1114f\") "
Oct 30 05:21:17.170523 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.170219    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-credential-socket" pod "e340979a-bec2-45fc-b742-604c80f1114f" (UID: "e340979a-bec2-45fc-b742-604c80f1114f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:21:17.170523 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.170227    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e340979a-bec2-45fc-b742-604c80f1114f-istiod-ca-cert" pod "e340979a-bec2-45fc-b742-604c80f1114f" (UID: "e340979a-bec2-45fc-b742-604c80f1114f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:21:17.170523 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.170348    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-workload-certs" pod "e340979a-bec2-45fc-b742-604c80f1114f" (UID: "e340979a-bec2-45fc-b742-604c80f1114f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:21:17.170699 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.170575    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-workload-socket" pod "e340979a-bec2-45fc-b742-604c80f1114f" (UID: "e340979a-bec2-45fc-b742-604c80f1114f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:21:17.170699 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.170662    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-istio-data" pod "e340979a-bec2-45fc-b742-604c80f1114f" (UID: "e340979a-bec2-45fc-b742-604c80f1114f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:21:17.170794 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.170729    4874 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-workload-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:21:17.170794 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.170748    4874 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e340979a-bec2-45fc-b742-604c80f1114f-istiod-ca-cert\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:21:17.170794 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.170763    4874 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-credential-socket\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:21:17.170794 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.170777    4874 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-workload-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:21:17.172113 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.172086    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-istio-envoy" pod "e340979a-bec2-45fc-b742-604c80f1114f" (UID: "e340979a-bec2-45fc-b742-604c80f1114f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:21:17.172299 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.172276    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e340979a-bec2-45fc-b742-604c80f1114f-istio-token" pod "e340979a-bec2-45fc-b742-604c80f1114f" (UID: "e340979a-bec2-45fc-b742-604c80f1114f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:21:17.172502 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.172487    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e340979a-bec2-45fc-b742-604c80f1114f-istio-podinfo" pod "e340979a-bec2-45fc-b742-604c80f1114f" (UID: "e340979a-bec2-45fc-b742-604c80f1114f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 05:21:17.172609 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.172594    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e340979a-bec2-45fc-b742-604c80f1114f-kube-api-access-s25nw" pod "e340979a-bec2-45fc-b742-604c80f1114f" (UID: "e340979a-bec2-45fc-b742-604c80f1114f"). InnerVolumeSpecName "kube-api-access-s25nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:21:17.271710 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.271676    4874 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-istio-data\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:21:17.271710 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.271705    4874 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e340979a-bec2-45fc-b742-604c80f1114f-istio-podinfo\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:21:17.271710 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.271715    4874 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e340979a-bec2-45fc-b742-604c80f1114f-istio-envoy\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:21:17.271956 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.271725    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s25nw\" (UniqueName: \"kubernetes.io/projected/e340979a-bec2-45fc-b742-604c80f1114f-kube-api-access-s25nw\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:21:17.271956 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.271734    4874 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e340979a-bec2-45fc-b742-604c80f1114f-istio-token\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:21:17.431020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.430989    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"]
Oct 30 05:21:17.442945 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:17.442920    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6949ad-openshift-default-698bf75589ngp7q"]
Oct 30 05:21:18.641472 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:18.641421    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7" podUID="4175bf4d-8756-44e5-8e2d-9c02dbd985f0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:21:18.654455 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:18.641431    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk" podUID="6141e561-11f5-42cb-90ec-680977c39d45" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:21:18.727578 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:18.727544    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e340979a-bec2-45fc-b742-604c80f1114f" path="/var/lib/kubelet/pods/e340979a-bec2-45fc-b742-604c80f1114f/volumes"
Oct 30 05:21:19.537193 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:19.537159    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:21:22.783360 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:22.783323    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fh868 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:21:22.783979 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:22.783943    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868" podUID="7059d28d-3eca-44fa-80c4-ac634a59aaf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:21:22.793270 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:22.792311    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-x8mpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:21:22.793270 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:22.792360    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd" podUID="c2ea846a-5ce0-424a-bb80-ec759b592c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:21:25.607912 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:21:25.607869    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:21:28.180069 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:28.180034    4874 generic.go:356] "Generic (PLEG): container finished" podID="6141e561-11f5-42cb-90ec-680977c39d45" containerID="809b687734b4127b9781ef8015227ebaf27f474ec4cbc8dd1d40501a8888ae5e" exitCode=0
Oct 30 05:21:28.180569 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:28.180110    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk" event={"ID":"6141e561-11f5-42cb-90ec-680977c39d45","Type":"ContainerDied","Data":"809b687734b4127b9781ef8015227ebaf27f474ec4cbc8dd1d40501a8888ae5e"}
Oct 30 05:21:28.182046 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:28.182020    4874 generic.go:356] "Generic (PLEG): container finished" podID="4175bf4d-8756-44e5-8e2d-9c02dbd985f0" containerID="63ef0f839a911129ca9ff817091791bbb375ae58114ee942a8aada35147b51e1" exitCode=0
Oct 30 05:21:28.182185 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:28.182077    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7" event={"ID":"4175bf4d-8756-44e5-8e2d-9c02dbd985f0","Type":"ContainerDied","Data":"63ef0f839a911129ca9ff817091791bbb375ae58114ee942a8aada35147b51e1"}
Oct 30 05:21:28.215666 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:28.215643    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk"
Oct 30 05:21:28.248644 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:28.248626    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7"
Oct 30 05:21:28.303013 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:28.302992    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4175bf4d-8756-44e5-8e2d-9c02dbd985f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4175bf4d-8756-44e5-8e2d-9c02dbd985f0-plugins-config-volume\") pod \"4175bf4d-8756-44e5-8e2d-9c02dbd985f0\" (UID: \"4175bf4d-8756-44e5-8e2d-9c02dbd985f0\") "
Oct 30 05:21:28.303134 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:28.303083    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6141e561-11f5-42cb-90ec-680977c39d45-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6141e561-11f5-42cb-90ec-680977c39d45-plugins-config-volume\") pod \"6141e561-11f5-42cb-90ec-680977c39d45\" (UID: \"6141e561-11f5-42cb-90ec-680977c39d45\") "
Oct 30 05:21:28.303181 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:28.303139    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4175bf4d-8756-44e5-8e2d-9c02dbd985f0-kube-api-access-bj58p\" (UniqueName: \"kubernetes.io/projected/4175bf4d-8756-44e5-8e2d-9c02dbd985f0-kube-api-access-bj58p\") pod \"4175bf4d-8756-44e5-8e2d-9c02dbd985f0\" (UID: \"4175bf4d-8756-44e5-8e2d-9c02dbd985f0\") "
Oct 30 05:21:28.303234 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:28.303176    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6141e561-11f5-42cb-90ec-680977c39d45-kube-api-access-d4d8h\" (UniqueName: \"kubernetes.io/projected/6141e561-11f5-42cb-90ec-680977c39d45-kube-api-access-d4d8h\") pod \"6141e561-11f5-42cb-90ec-680977c39d45\" (UID: \"6141e561-11f5-42cb-90ec-680977c39d45\") "
Oct 30 05:21:28.303447 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:28.303425    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4175bf4d-8756-44e5-8e2d-9c02dbd985f0-plugins-config-volume" pod "4175bf4d-8756-44e5-8e2d-9c02dbd985f0" (UID: "4175bf4d-8756-44e5-8e2d-9c02dbd985f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:21:28.303530 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:28.303506    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6141e561-11f5-42cb-90ec-680977c39d45-plugins-config-volume" pod "6141e561-11f5-42cb-90ec-680977c39d45" (UID: "6141e561-11f5-42cb-90ec-680977c39d45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:21:28.305166 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:28.305144    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6141e561-11f5-42cb-90ec-680977c39d45-kube-api-access-d4d8h" pod "6141e561-11f5-42cb-90ec-680977c39d45" (UID: "6141e561-11f5-42cb-90ec-680977c39d45"). InnerVolumeSpecName "kube-api-access-d4d8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:21:28.305224 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:28.305176    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4175bf4d-8756-44e5-8e2d-9c02dbd985f0-kube-api-access-bj58p" pod "4175bf4d-8756-44e5-8e2d-9c02dbd985f0" (UID: "4175bf4d-8756-44e5-8e2d-9c02dbd985f0"). InnerVolumeSpecName "kube-api-access-bj58p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:21:28.404113 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:28.404083    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6141e561-11f5-42cb-90ec-680977c39d45-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:21:28.404113 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:28.404109    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj58p\" (UniqueName: \"kubernetes.io/projected/4175bf4d-8756-44e5-8e2d-9c02dbd985f0-kube-api-access-bj58p\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:21:28.404277 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:28.404123    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4d8h\" (UniqueName: \"kubernetes.io/projected/6141e561-11f5-42cb-90ec-680977c39d45-kube-api-access-d4d8h\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:21:28.404277 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:28.404135    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4175bf4d-8756-44e5-8e2d-9c02dbd985f0-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:21:29.187474 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:29.187389    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk"
Oct 30 05:21:29.187474 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:29.187429    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk" event={"ID":"6141e561-11f5-42cb-90ec-680977c39d45","Type":"ContainerDied","Data":"7846e5070704bdb59708bc02c7b08d1dfcbb049a79a1306ba862041c14f57224"}
Oct 30 05:21:29.187978 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:29.187488    4874 scope.go:122] "RemoveContainer" containerID="809b687734b4127b9781ef8015227ebaf27f474ec4cbc8dd1d40501a8888ae5e"
Oct 30 05:21:29.188943 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:29.188922    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7"
Oct 30 05:21:29.189029 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:29.188921    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7" event={"ID":"4175bf4d-8756-44e5-8e2d-9c02dbd985f0","Type":"ContainerDied","Data":"5715a9d11c69fda61db66e31780a55352ba2e0cd9d128996f729fc8041a7bf33"}
Oct 30 05:21:29.195887 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:29.195868    4874 scope.go:122] "RemoveContainer" containerID="63ef0f839a911129ca9ff817091791bbb375ae58114ee942a8aada35147b51e1"
Oct 30 05:21:29.238598 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:29.238574    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk"]
Oct 30 05:21:29.249369 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:29.249345    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6949ad-558767ff7n6snk"]
Oct 30 05:21:29.283181 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:29.283162    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7"]
Oct 30 05:21:29.290466 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:29.290446    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6949ad-8558cg88f7"]
Oct 30 05:21:30.727095 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:30.727059    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4175bf4d-8756-44e5-8e2d-9c02dbd985f0" path="/var/lib/kubelet/pods/4175bf4d-8756-44e5-8e2d-9c02dbd985f0/volumes"
Oct 30 05:21:30.727491 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:30.727473    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6141e561-11f5-42cb-90ec-680977c39d45" path="/var/lib/kubelet/pods/6141e561-11f5-42cb-90ec-680977c39d45/volumes"
Oct 30 05:21:33.936955 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:33.936918    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"]
Oct 30 05:21:33.937425 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:33.937300    4874 cpu_manager.go:414] "containerMap: removing container" podUID="0de9acc3-a43d-4bde-9cc1-aa028192db23" containerName="maas-api"
Oct 30 05:21:33.937425 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:33.937312    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0de9acc3-a43d-4bde-9cc1-aa028192db23" containerName="maas-api"
Oct 30 05:21:33.937425 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:33.937330    4874 cpu_manager.go:414] "containerMap: removing container" podUID="e340979a-bec2-45fc-b742-604c80f1114f" containerName="istio-proxy"
Oct 30 05:21:33.937425 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:33.937335    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e340979a-bec2-45fc-b742-604c80f1114f" containerName="istio-proxy"
Oct 30 05:21:33.937425 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:33.937344    4874 cpu_manager.go:414] "containerMap: removing container" podUID="6141e561-11f5-42cb-90ec-680977c39d45" containerName="payload-processing"
Oct 30 05:21:33.937425 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:33.937350    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6141e561-11f5-42cb-90ec-680977c39d45" containerName="payload-processing"
Oct 30 05:21:33.937425 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:33.937366    4874 cpu_manager.go:414] "containerMap: removing container" podUID="4175bf4d-8756-44e5-8e2d-9c02dbd985f0" containerName="payload-pre-processing"
Oct 30 05:21:33.937425 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:33.937374    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4175bf4d-8756-44e5-8e2d-9c02dbd985f0" containerName="payload-pre-processing"
Oct 30 05:21:33.937425 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:33.937382    4874 cpu_manager.go:414] "containerMap: removing container" podUID="e340979a-bec2-45fc-b742-604c80f1114f" containerName="istio-proxy"
Oct 30 05:21:33.937425 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:33.937386    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e340979a-bec2-45fc-b742-604c80f1114f" containerName="istio-proxy"
Oct 30 05:21:33.937425 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:33.937393    4874 cpu_manager.go:414] "containerMap: removing container" podUID="b259f029-bdc1-4880-aeb7-cc3a47c8216b" containerName="main"
Oct 30 05:21:33.937425 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:33.937397    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b259f029-bdc1-4880-aeb7-cc3a47c8216b" containerName="main"
Oct 30 05:21:33.937811 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:33.937446    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="6141e561-11f5-42cb-90ec-680977c39d45" containerName="payload-processing"
Oct 30 05:21:33.937811 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:33.937458    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="4175bf4d-8756-44e5-8e2d-9c02dbd985f0" containerName="payload-pre-processing"
Oct 30 05:21:33.937811 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:33.937465    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="e340979a-bec2-45fc-b742-604c80f1114f" containerName="istio-proxy"
Oct 30 05:21:33.937811 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:33.937471    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="b259f029-bdc1-4880-aeb7-cc3a47c8216b" containerName="main"
Oct 30 05:21:33.937811 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:33.937477    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="0de9acc3-a43d-4bde-9cc1-aa028192db23" containerName="maas-api"
Oct 30 05:21:33.941498 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:33.941481    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:21:33.967095 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:33.967075    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-d1402746-kserve-self-signed-certs\""
Oct 30 05:21:33.983536 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:33.983508    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"]
Oct 30 05:21:34.049477 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:34.049445    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-home\") pod \"e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k\" (UID: \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\") " pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:21:34.049638 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:34.049483    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-tmp-dir\") pod \"e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k\" (UID: \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\") " pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:21:34.049638 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:34.049560    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be25aad-259c-43e9-aef5-e37e1d0c3f47-tls-certs\") pod \"e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k\" (UID: \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\") " pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:21:34.049638 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:34.049618    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-model-cache\") pod \"e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k\" (UID: \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\") " pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:21:34.050035 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:34.049667    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt45s\" (UniqueName: \"kubernetes.io/projected/6be25aad-259c-43e9-aef5-e37e1d0c3f47-kube-api-access-rt45s\") pod \"e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k\" (UID: \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\") " pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:21:34.050035 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:34.049720    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-dshm\") pod \"e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k\" (UID: \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\") " pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:21:34.150606 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:34.150553    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-dshm\") pod \"e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k\" (UID: \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\") " pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:21:34.150813 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:34.150636    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-home\") pod \"e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k\" (UID: \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\") " pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:21:34.150813 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:34.150666    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-tmp-dir\") pod \"e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k\" (UID: \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\") " pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:21:34.151084 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:34.151055    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-tmp-dir\") pod \"e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k\" (UID: \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\") " pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:21:34.151084 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:34.151074    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-home\") pod \"e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k\" (UID: \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\") " pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:21:34.151331 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:34.151221    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be25aad-259c-43e9-aef5-e37e1d0c3f47-tls-certs\") pod \"e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k\" (UID: \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\") " pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:21:34.151331 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:34.151308    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-model-cache\") pod \"e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k\" (UID: \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\") " pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:21:34.151450 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:34.151368    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt45s\" (UniqueName: \"kubernetes.io/projected/6be25aad-259c-43e9-aef5-e37e1d0c3f47-kube-api-access-rt45s\") pod \"e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k\" (UID: \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\") " pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:21:34.151570 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:34.151551    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-model-cache\") pod \"e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k\" (UID: \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\") " pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:21:34.152949 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:34.152928    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-dshm\") pod \"e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k\" (UID: \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\") " pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:21:34.153276 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:34.153260    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be25aad-259c-43e9-aef5-e37e1d0c3f47-tls-certs\") pod \"e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k\" (UID: \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\") " pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:21:34.174886 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:34.174859    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt45s\" (UniqueName: \"kubernetes.io/projected/6be25aad-259c-43e9-aef5-e37e1d0c3f47-kube-api-access-rt45s\") pod \"e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k\" (UID: \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\") " pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:21:34.253110 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:34.253078    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:21:34.556155 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:21:34.556127    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6be25aad_259c_43e9_aef5_e37e1d0c3f47.slice/crio-7a83b0cb716845bc128db06493951b679f1c50d10c4cb50d2f1a2ac5fd68d2fa WatchSource:0}: Error finding container 7a83b0cb716845bc128db06493951b679f1c50d10c4cb50d2f1a2ac5fd68d2fa: Status 404 returned error can't find the container with id 7a83b0cb716845bc128db06493951b679f1c50d10c4cb50d2f1a2ac5fd68d2fa
Oct 30 05:21:34.556297 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:34.556150    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"]
Oct 30 05:21:35.218572 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:35.218538    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k" event={"ID":"6be25aad-259c-43e9-aef5-e37e1d0c3f47","Type":"ContainerStarted","Data":"0eb9d78bed094de5972c77fb09e8d82af68bb75a3adc76d6f311a713b882a59e"}
Oct 30 05:21:35.218966 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:35.218577    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k" event={"ID":"6be25aad-259c-43e9-aef5-e37e1d0c3f47","Type":"ContainerStarted","Data":"7a83b0cb716845bc128db06493951b679f1c50d10c4cb50d2f1a2ac5fd68d2fa"}
Oct 30 05:21:35.253360 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:35.253316    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:21:35.267948 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:35.267903    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k" podStartSLOduration=2.267889857 podStartE2EDuration="2.267889857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:21:35.266418133 +0000 UTC m=+5014.678813705" watchObservedRunningTime="2026-10-30 05:21:35.267889857 +0000 UTC m=+5014.680285430"
Oct 30 05:21:36.563909 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:36.563875    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:21:44.253977 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:44.253940    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:21:44.266666 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:44.266642    4874 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:21:44.295508 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:44.295483    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:21:50.045899 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:50.045866    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m25pc"]
Oct 30 05:21:50.067207 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:50.067176    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m25pc"]
Oct 30 05:21:50.727975 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:50.727940    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d6a1216-a804-43f9-b5bf-60d9c511df94" path="/var/lib/kubelet/pods/2d6a1216-a804-43f9-b5bf-60d9c511df94/volumes"
Oct 30 05:21:55.608372 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:21:55.608331    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:21:56.754775 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:56.754743    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:21:57.546781 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:57.546747    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lr5lm"]
Oct 30 05:21:57.547500 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:57.547474    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="e340979a-bec2-45fc-b742-604c80f1114f" containerName="istio-proxy"
Oct 30 05:21:57.551119 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:57.551096    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lr5lm"
Oct 30 05:21:57.574691 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:57.574663    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lr5lm"]
Oct 30 05:21:57.583972 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:57.583942    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:21:57.584161 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:57.584030    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:21:57.645537 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:57.645503    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003a071c-7125-4bd6-964a-8e744bab41ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-lr5lm\" (UID: \"003a071c-7125-4bd6-964a-8e744bab41ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lr5lm"
Oct 30 05:21:57.645730 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:57.645555    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7nhx\" (UniqueName: \"kubernetes.io/projected/003a071c-7125-4bd6-964a-8e744bab41ae-kube-api-access-l7nhx\") pod \"maas-api-revoke-sub-452b6d7b-lr5lm\" (UID: \"003a071c-7125-4bd6-964a-8e744bab41ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lr5lm"
Oct 30 05:21:57.645730 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:57.645584    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003a071c-7125-4bd6-964a-8e744bab41ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-lr5lm\" (UID: \"003a071c-7125-4bd6-964a-8e744bab41ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lr5lm"
Oct 30 05:21:57.747188 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:57.747149    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003a071c-7125-4bd6-964a-8e744bab41ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-lr5lm\" (UID: \"003a071c-7125-4bd6-964a-8e744bab41ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lr5lm"
Oct 30 05:21:57.747399 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:57.747205    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7nhx\" (UniqueName: \"kubernetes.io/projected/003a071c-7125-4bd6-964a-8e744bab41ae-kube-api-access-l7nhx\") pod \"maas-api-revoke-sub-452b6d7b-lr5lm\" (UID: \"003a071c-7125-4bd6-964a-8e744bab41ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lr5lm"
Oct 30 05:21:57.747561 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:57.747536    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003a071c-7125-4bd6-964a-8e744bab41ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-lr5lm\" (UID: \"003a071c-7125-4bd6-964a-8e744bab41ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lr5lm"
Oct 30 05:21:57.748401 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:57.748371    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003a071c-7125-4bd6-964a-8e744bab41ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-lr5lm\" (UID: \"003a071c-7125-4bd6-964a-8e744bab41ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lr5lm"
Oct 30 05:21:57.759660 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:57.759638    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003a071c-7125-4bd6-964a-8e744bab41ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-lr5lm\" (UID: \"003a071c-7125-4bd6-964a-8e744bab41ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lr5lm"
Oct 30 05:21:57.778353 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:57.778327    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7nhx\" (UniqueName: \"kubernetes.io/projected/003a071c-7125-4bd6-964a-8e744bab41ae-kube-api-access-l7nhx\") pod \"maas-api-revoke-sub-452b6d7b-lr5lm\" (UID: \"003a071c-7125-4bd6-964a-8e744bab41ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lr5lm"
Oct 30 05:21:57.869894 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:57.869802    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lr5lm"
Oct 30 05:21:58.198837 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:58.198739    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lr5lm"]
Oct 30 05:21:58.362262 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:58.353546    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lr5lm" event={"ID":"003a071c-7125-4bd6-964a-8e744bab41ae","Type":"ContainerStarted","Data":"d7de04ad96ea31eb2231f09d6616666c15e4254a479e40b58bbb8ef6538c73e7"}
Oct 30 05:21:59.369727 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:59.369325    4874 generic.go:356] "Generic (PLEG): container finished" podID="003a071c-7125-4bd6-964a-8e744bab41ae" containerID="3e053b3fa02751ec05a7756804677d67aac3795320d9167c059f07171bd9a818" exitCode=0
Oct 30 05:21:59.369727 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:21:59.369414    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lr5lm" event={"ID":"003a071c-7125-4bd6-964a-8e744bab41ae","Type":"ContainerDied","Data":"3e053b3fa02751ec05a7756804677d67aac3795320d9167c059f07171bd9a818"}
Oct 30 05:22:00.734767 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:00.734740    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lr5lm"
Oct 30 05:22:00.808438 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:00.808412    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/003a071c-7125-4bd6-964a-8e744bab41ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003a071c-7125-4bd6-964a-8e744bab41ae-maas-api-xks-ca\") pod \"003a071c-7125-4bd6-964a-8e744bab41ae\" (UID: \"003a071c-7125-4bd6-964a-8e744bab41ae\") "
Oct 30 05:22:00.808625 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:00.808456    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/003a071c-7125-4bd6-964a-8e744bab41ae-kube-api-access-l7nhx\" (UniqueName: \"kubernetes.io/projected/003a071c-7125-4bd6-964a-8e744bab41ae-kube-api-access-l7nhx\") pod \"003a071c-7125-4bd6-964a-8e744bab41ae\" (UID: \"003a071c-7125-4bd6-964a-8e744bab41ae\") "
Oct 30 05:22:00.808625 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:00.808506    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/003a071c-7125-4bd6-964a-8e744bab41ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003a071c-7125-4bd6-964a-8e744bab41ae-maas-api-openshift-ca\") pod \"003a071c-7125-4bd6-964a-8e744bab41ae\" (UID: \"003a071c-7125-4bd6-964a-8e744bab41ae\") "
Oct 30 05:22:00.809012 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:00.808946    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003a071c-7125-4bd6-964a-8e744bab41ae-maas-api-openshift-ca" pod "003a071c-7125-4bd6-964a-8e744bab41ae" (UID: "003a071c-7125-4bd6-964a-8e744bab41ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:22:00.809539 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:00.809507    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003a071c-7125-4bd6-964a-8e744bab41ae-maas-api-openshift-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:22:00.810709 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:00.810689    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/003a071c-7125-4bd6-964a-8e744bab41ae-maas-api-xks-ca" pod "003a071c-7125-4bd6-964a-8e744bab41ae" (UID: "003a071c-7125-4bd6-964a-8e744bab41ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:22:00.811006 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:00.810991    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003a071c-7125-4bd6-964a-8e744bab41ae-kube-api-access-l7nhx" pod "003a071c-7125-4bd6-964a-8e744bab41ae" (UID: "003a071c-7125-4bd6-964a-8e744bab41ae"). InnerVolumeSpecName "kube-api-access-l7nhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:22:00.910316 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:00.910292    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003a071c-7125-4bd6-964a-8e744bab41ae-maas-api-xks-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:22:00.910316 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:00.910315    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7nhx\" (UniqueName: \"kubernetes.io/projected/003a071c-7125-4bd6-964a-8e744bab41ae-kube-api-access-l7nhx\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:22:01.390903 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:01.390864    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lr5lm"
Oct 30 05:22:01.391054 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:01.390904    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lr5lm" event={"ID":"003a071c-7125-4bd6-964a-8e744bab41ae","Type":"ContainerDied","Data":"d7de04ad96ea31eb2231f09d6616666c15e4254a479e40b58bbb8ef6538c73e7"}
Oct 30 05:22:01.391054 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:01.390997    4874 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7de04ad96ea31eb2231f09d6616666c15e4254a479e40b58bbb8ef6538c73e7"
Oct 30 05:22:11.139622 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:11.139586    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:22:14.200265 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:14.186165    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"]
Oct 30 05:22:18.728328 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:18.728293    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fmrhp"]
Oct 30 05:22:18.728820 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:18.728798    4874 cpu_manager.go:414] "containerMap: removing container" podUID="003a071c-7125-4bd6-964a-8e744bab41ae" containerName="revoke-keys"
Oct 30 05:22:18.728820 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:18.728817    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="003a071c-7125-4bd6-964a-8e744bab41ae" containerName="revoke-keys"
Oct 30 05:22:18.728958 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:18.728941    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="003a071c-7125-4bd6-964a-8e744bab41ae" containerName="revoke-keys"
Oct 30 05:22:18.731799 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:18.731779    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fmrhp"
Oct 30 05:22:18.747212 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:18.747179    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fmrhp"]
Oct 30 05:22:18.755934 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:18.755900    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:22:18.755934 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:18.755915    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:22:18.802599 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:18.802566    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15cef83-2199-4538-98dd-adda41c623a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-fmrhp\" (UID: \"a15cef83-2199-4538-98dd-adda41c623a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fmrhp"
Oct 30 05:22:18.802770 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:18.802621    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5pf5\" (UniqueName: \"kubernetes.io/projected/a15cef83-2199-4538-98dd-adda41c623a2-kube-api-access-r5pf5\") pod \"maas-api-revoke-sub-3e31a506-fmrhp\" (UID: \"a15cef83-2199-4538-98dd-adda41c623a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fmrhp"
Oct 30 05:22:18.802770 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:18.802698    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15cef83-2199-4538-98dd-adda41c623a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-fmrhp\" (UID: \"a15cef83-2199-4538-98dd-adda41c623a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fmrhp"
Oct 30 05:22:18.903539 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:18.903508    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5pf5\" (UniqueName: \"kubernetes.io/projected/a15cef83-2199-4538-98dd-adda41c623a2-kube-api-access-r5pf5\") pod \"maas-api-revoke-sub-3e31a506-fmrhp\" (UID: \"a15cef83-2199-4538-98dd-adda41c623a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fmrhp"
Oct 30 05:22:18.903716 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:18.903553    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15cef83-2199-4538-98dd-adda41c623a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-fmrhp\" (UID: \"a15cef83-2199-4538-98dd-adda41c623a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fmrhp"
Oct 30 05:22:18.903716 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:18.903619    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15cef83-2199-4538-98dd-adda41c623a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-fmrhp\" (UID: \"a15cef83-2199-4538-98dd-adda41c623a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fmrhp"
Oct 30 05:22:18.904262 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:18.904226    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15cef83-2199-4538-98dd-adda41c623a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-fmrhp\" (UID: \"a15cef83-2199-4538-98dd-adda41c623a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fmrhp"
Oct 30 05:22:18.905930 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:18.905904    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15cef83-2199-4538-98dd-adda41c623a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-fmrhp\" (UID: \"a15cef83-2199-4538-98dd-adda41c623a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fmrhp"
Oct 30 05:22:18.926257 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:18.926221    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5pf5\" (UniqueName: \"kubernetes.io/projected/a15cef83-2199-4538-98dd-adda41c623a2-kube-api-access-r5pf5\") pod \"maas-api-revoke-sub-3e31a506-fmrhp\" (UID: \"a15cef83-2199-4538-98dd-adda41c623a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fmrhp"
Oct 30 05:22:19.047217 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:19.047151    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fmrhp"
Oct 30 05:22:19.353788 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:19.353715    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fmrhp"]
Oct 30 05:22:19.357132 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:22:19.357101    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15cef83_2199_4538_98dd_adda41c623a2.slice/crio-7d82f15fcf26c50857e7fdcf16af1e5476f8d6307601a2ffcd2b66c32f5da77c WatchSource:0}: Error finding container 7d82f15fcf26c50857e7fdcf16af1e5476f8d6307601a2ffcd2b66c32f5da77c: Status 404 returned error can't find the container with id 7d82f15fcf26c50857e7fdcf16af1e5476f8d6307601a2ffcd2b66c32f5da77c
Oct 30 05:22:19.546604 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:19.546573    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fmrhp" event={"ID":"a15cef83-2199-4538-98dd-adda41c623a2","Type":"ContainerStarted","Data":"7d82f15fcf26c50857e7fdcf16af1e5476f8d6307601a2ffcd2b66c32f5da77c"}
Oct 30 05:22:19.991370 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:19.991337    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:22:20.555138 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:20.555100    4874 generic.go:356] "Generic (PLEG): container finished" podID="a15cef83-2199-4538-98dd-adda41c623a2" containerID="dd1bd577ba5e84d62b9314b09d9b03791884e43a40139efdf27a32b50b2a486e" exitCode=0
Oct 30 05:22:20.555374 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:20.555152    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fmrhp" event={"ID":"a15cef83-2199-4538-98dd-adda41c623a2","Type":"ContainerDied","Data":"dd1bd577ba5e84d62b9314b09d9b03791884e43a40139efdf27a32b50b2a486e"}
Oct 30 05:22:22.231699 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:22.231404    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fmrhp"
Oct 30 05:22:22.366263 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:22.364891    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a15cef83-2199-4538-98dd-adda41c623a2-kube-api-access-r5pf5\" (UniqueName: \"kubernetes.io/projected/a15cef83-2199-4538-98dd-adda41c623a2-kube-api-access-r5pf5\") pod \"a15cef83-2199-4538-98dd-adda41c623a2\" (UID: \"a15cef83-2199-4538-98dd-adda41c623a2\") "
Oct 30 05:22:22.366263 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:22.364954    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a15cef83-2199-4538-98dd-adda41c623a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15cef83-2199-4538-98dd-adda41c623a2-maas-api-xks-ca\") pod \"a15cef83-2199-4538-98dd-adda41c623a2\" (UID: \"a15cef83-2199-4538-98dd-adda41c623a2\") "
Oct 30 05:22:22.366263 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:22.365027    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a15cef83-2199-4538-98dd-adda41c623a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15cef83-2199-4538-98dd-adda41c623a2-maas-api-openshift-ca\") pod \"a15cef83-2199-4538-98dd-adda41c623a2\" (UID: \"a15cef83-2199-4538-98dd-adda41c623a2\") "
Oct 30 05:22:22.381021 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:22.380991    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a15cef83-2199-4538-98dd-adda41c623a2-maas-api-openshift-ca" pod "a15cef83-2199-4538-98dd-adda41c623a2" (UID: "a15cef83-2199-4538-98dd-adda41c623a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:22:22.404769 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:22.404740    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15cef83-2199-4538-98dd-adda41c623a2-kube-api-access-r5pf5" pod "a15cef83-2199-4538-98dd-adda41c623a2" (UID: "a15cef83-2199-4538-98dd-adda41c623a2"). InnerVolumeSpecName "kube-api-access-r5pf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:22:22.434845 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:22.434807    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a15cef83-2199-4538-98dd-adda41c623a2-maas-api-xks-ca" pod "a15cef83-2199-4538-98dd-adda41c623a2" (UID: "a15cef83-2199-4538-98dd-adda41c623a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:22:22.469112 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:22.468838    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5pf5\" (UniqueName: \"kubernetes.io/projected/a15cef83-2199-4538-98dd-adda41c623a2-kube-api-access-r5pf5\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:22:22.469112 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:22.468891    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15cef83-2199-4538-98dd-adda41c623a2-maas-api-xks-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:22:22.469112 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:22.468908    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15cef83-2199-4538-98dd-adda41c623a2-maas-api-openshift-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:22:22.591314 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:22.590803    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fmrhp" event={"ID":"a15cef83-2199-4538-98dd-adda41c623a2","Type":"ContainerDied","Data":"7d82f15fcf26c50857e7fdcf16af1e5476f8d6307601a2ffcd2b66c32f5da77c"}
Oct 30 05:22:22.591314 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:22.590871    4874 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d82f15fcf26c50857e7fdcf16af1e5476f8d6307601a2ffcd2b66c32f5da77c"
Oct 30 05:22:22.591314 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:22.590943    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fmrhp"
Oct 30 05:22:22.786281 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:22.783322    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-x8mpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:22:22.786281 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:22.783391    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd" podUID="c2ea846a-5ce0-424a-bb80-ec759b592c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:22:22.790273 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:22.787309    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fh868 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:22:22.790273 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:22.787355    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868" podUID="7059d28d-3eca-44fa-80c4-ac634a59aaf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:22:25.608519 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:22:25.608484    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:22:29.514516 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:29.514458    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k" podUID="6be25aad-259c-43e9-aef5-e37e1d0c3f47" containerName="main" containerID="cri-o://0eb9d78bed094de5972c77fb09e8d82af68bb75a3adc76d6f311a713b882a59e" gracePeriod=45
Oct 30 05:22:29.760654 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:29.760630    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:22:29.846224 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:29.846146    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-home\" (UniqueName: \"kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-home\") pod \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\" (UID: \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\") "
Oct 30 05:22:29.846224 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:29.846200    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-model-cache\") pod \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\" (UID: \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\") "
Oct 30 05:22:29.846509 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:29.846260    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6be25aad-259c-43e9-aef5-e37e1d0c3f47-tls-certs\" (UniqueName: \"kubernetes.io/secret/6be25aad-259c-43e9-aef5-e37e1d0c3f47-tls-certs\") pod \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\" (UID: \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\") "
Oct 30 05:22:29.846509 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:29.846297    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-dshm\") pod \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\" (UID: \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\") "
Oct 30 05:22:29.846509 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:29.846373    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be25aad-259c-43e9-aef5-e37e1d0c3f47-kube-api-access-rt45s\" (UniqueName: \"kubernetes.io/projected/6be25aad-259c-43e9-aef5-e37e1d0c3f47-kube-api-access-rt45s\") pod \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\" (UID: \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\") "
Oct 30 05:22:29.846509 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:29.846408    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-tmp-dir\") pod \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\" (UID: \"6be25aad-259c-43e9-aef5-e37e1d0c3f47\") "
Oct 30 05:22:29.846747 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:29.846458    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-home" pod "6be25aad-259c-43e9-aef5-e37e1d0c3f47" (UID: "6be25aad-259c-43e9-aef5-e37e1d0c3f47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:22:29.846747 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:29.846526    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-model-cache" pod "6be25aad-259c-43e9-aef5-e37e1d0c3f47" (UID: "6be25aad-259c-43e9-aef5-e37e1d0c3f47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:22:29.846747 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:29.846724    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-tmp-dir" pod "6be25aad-259c-43e9-aef5-e37e1d0c3f47" (UID: "6be25aad-259c-43e9-aef5-e37e1d0c3f47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:22:29.846927 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:29.846753    4874 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-home\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:22:29.846927 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:29.846771    4874 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-model-cache\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:22:29.848644 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:29.848617    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be25aad-259c-43e9-aef5-e37e1d0c3f47-tls-certs" pod "6be25aad-259c-43e9-aef5-e37e1d0c3f47" (UID: "6be25aad-259c-43e9-aef5-e37e1d0c3f47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:22:29.848644 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:29.848623    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-dshm" pod "6be25aad-259c-43e9-aef5-e37e1d0c3f47" (UID: "6be25aad-259c-43e9-aef5-e37e1d0c3f47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 05:22:29.848831 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:29.848652    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be25aad-259c-43e9-aef5-e37e1d0c3f47-kube-api-access-rt45s" pod "6be25aad-259c-43e9-aef5-e37e1d0c3f47" (UID: "6be25aad-259c-43e9-aef5-e37e1d0c3f47"). InnerVolumeSpecName "kube-api-access-rt45s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:22:29.948065 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:29.948040    4874 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-dshm\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:22:29.948065 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:29.948063    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt45s\" (UniqueName: \"kubernetes.io/projected/6be25aad-259c-43e9-aef5-e37e1d0c3f47-kube-api-access-rt45s\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:22:29.948220 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:29.948092    4874 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be25aad-259c-43e9-aef5-e37e1d0c3f47-tmp-dir\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:22:29.948220 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:29.948100    4874 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be25aad-259c-43e9-aef5-e37e1d0c3f47-tls-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:22:30.284202 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:30.284164    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:22:30.633484 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:30.633387    4874 generic.go:356] "Generic (PLEG): container finished" podID="6be25aad-259c-43e9-aef5-e37e1d0c3f47" containerID="0eb9d78bed094de5972c77fb09e8d82af68bb75a3adc76d6f311a713b882a59e" exitCode=0
Oct 30 05:22:30.633974 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:30.633518    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k" event={"ID":"6be25aad-259c-43e9-aef5-e37e1d0c3f47","Type":"ContainerDied","Data":"0eb9d78bed094de5972c77fb09e8d82af68bb75a3adc76d6f311a713b882a59e"}
Oct 30 05:22:30.633974 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:30.633552    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k" event={"ID":"6be25aad-259c-43e9-aef5-e37e1d0c3f47","Type":"ContainerDied","Data":"7a83b0cb716845bc128db06493951b679f1c50d10c4cb50d2f1a2ac5fd68d2fa"}
Oct 30 05:22:30.633974 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:30.633574    4874 scope.go:122] "RemoveContainer" containerID="0eb9d78bed094de5972c77fb09e8d82af68bb75a3adc76d6f311a713b882a59e"
Oct 30 05:22:30.633974 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:30.633729    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"
Oct 30 05:22:30.649101 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:30.649071    4874 scope.go:122] "RemoveContainer" containerID="0eb9d78bed094de5972c77fb09e8d82af68bb75a3adc76d6f311a713b882a59e"
Oct 30 05:22:30.649533 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:22:30.649506    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eb9d78bed094de5972c77fb09e8d82af68bb75a3adc76d6f311a713b882a59e\": container with ID starting with 0eb9d78bed094de5972c77fb09e8d82af68bb75a3adc76d6f311a713b882a59e not found: ID does not exist" containerID="0eb9d78bed094de5972c77fb09e8d82af68bb75a3adc76d6f311a713b882a59e"
Oct 30 05:22:30.649692 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:30.649541    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb9d78bed094de5972c77fb09e8d82af68bb75a3adc76d6f311a713b882a59e"} err="failed to get container status \"0eb9d78bed094de5972c77fb09e8d82af68bb75a3adc76d6f311a713b882a59e\": rpc error: code = NotFound desc = could not find container \"0eb9d78bed094de5972c77fb09e8d82af68bb75a3adc76d6f311a713b882a59e\": container with ID starting with 0eb9d78bed094de5972c77fb09e8d82af68bb75a3adc76d6f311a713b882a59e not found: ID does not exist"
Oct 30 05:22:30.717005 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:30.716969    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"]
Oct 30 05:22:30.729144 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:30.729113    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-d1402746-kserve-6d7cddc457-kgq2k"]
Oct 30 05:22:32.727672 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:32.727643    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be25aad-259c-43e9-aef5-e37e1d0c3f47" path="/var/lib/kubelet/pods/6be25aad-259c-43e9-aef5-e37e1d0c3f47/volumes"
Oct 30 05:22:35.032475 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:35.032440    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:22:37.784435 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:37.784390    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fh868 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:22:37.785098 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:37.785064    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fh868" podUID="7059d28d-3eca-44fa-80c4-ac634a59aaf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:22:37.789446 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:37.789353    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-x8mpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:22:37.789446 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:37.789398    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-x8mpd" podUID="c2ea846a-5ce0-424a-bb80-ec759b592c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:22:39.036488 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:39.036458    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43eba674-9kf47"]
Oct 30 05:22:39.036903 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:39.036821    4874 cpu_manager.go:414] "containerMap: removing container" podUID="a15cef83-2199-4538-98dd-adda41c623a2" containerName="revoke-keys"
Oct 30 05:22:39.036903 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:39.036836    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15cef83-2199-4538-98dd-adda41c623a2" containerName="revoke-keys"
Oct 30 05:22:39.036903 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:39.036857    4874 cpu_manager.go:414] "containerMap: removing container" podUID="6be25aad-259c-43e9-aef5-e37e1d0c3f47" containerName="main"
Oct 30 05:22:39.036903 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:39.036863    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be25aad-259c-43e9-aef5-e37e1d0c3f47" containerName="main"
Oct 30 05:22:39.037051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:39.036927    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be25aad-259c-43e9-aef5-e37e1d0c3f47" containerName="main"
Oct 30 05:22:39.037051 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:39.036940    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15cef83-2199-4538-98dd-adda41c623a2" containerName="revoke-keys"
Oct 30 05:22:39.041138 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:39.041118    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43eba674-9kf47"
Oct 30 05:22:39.062659 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:39.062633    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:22:39.073466 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:39.073449    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:22:39.144862 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:39.144835    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43eba674-9kf47"]
Oct 30 05:22:39.169161 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:39.169132    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4378744-63f2-4b39-a8eb-e32a343b4ade-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43eba674-9kf47\" (UID: \"a4378744-63f2-4b39-a8eb-e32a343b4ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43eba674-9kf47"
Oct 30 05:22:39.169303 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:39.169188    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4378744-63f2-4b39-a8eb-e32a343b4ade-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43eba674-9kf47\" (UID: \"a4378744-63f2-4b39-a8eb-e32a343b4ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43eba674-9kf47"
Oct 30 05:22:39.169303 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:39.169283    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcznf\" (UniqueName: \"kubernetes.io/projected/a4378744-63f2-4b39-a8eb-e32a343b4ade-kube-api-access-mcznf\") pod \"maas-api-revoke-sub-43eba674-9kf47\" (UID: \"a4378744-63f2-4b39-a8eb-e32a343b4ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43eba674-9kf47"
Oct 30 05:22:39.270398 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:39.270370    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4378744-63f2-4b39-a8eb-e32a343b4ade-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43eba674-9kf47\" (UID: \"a4378744-63f2-4b39-a8eb-e32a343b4ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43eba674-9kf47"
Oct 30 05:22:39.270523 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:39.270411    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcznf\" (UniqueName: \"kubernetes.io/projected/a4378744-63f2-4b39-a8eb-e32a343b4ade-kube-api-access-mcznf\") pod \"maas-api-revoke-sub-43eba674-9kf47\" (UID: \"a4378744-63f2-4b39-a8eb-e32a343b4ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43eba674-9kf47"
Oct 30 05:22:39.270523 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:39.270464    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4378744-63f2-4b39-a8eb-e32a343b4ade-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43eba674-9kf47\" (UID: \"a4378744-63f2-4b39-a8eb-e32a343b4ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43eba674-9kf47"
Oct 30 05:22:39.271071 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:39.271050    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4378744-63f2-4b39-a8eb-e32a343b4ade-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43eba674-9kf47\" (UID: \"a4378744-63f2-4b39-a8eb-e32a343b4ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43eba674-9kf47"
Oct 30 05:22:39.272792 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:39.272769    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4378744-63f2-4b39-a8eb-e32a343b4ade-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43eba674-9kf47\" (UID: \"a4378744-63f2-4b39-a8eb-e32a343b4ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43eba674-9kf47"
Oct 30 05:22:39.295701 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:39.295650    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcznf\" (UniqueName: \"kubernetes.io/projected/a4378744-63f2-4b39-a8eb-e32a343b4ade-kube-api-access-mcznf\") pod \"maas-api-revoke-sub-43eba674-9kf47\" (UID: \"a4378744-63f2-4b39-a8eb-e32a343b4ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43eba674-9kf47"
Oct 30 05:22:39.353612 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:39.353589    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43eba674-9kf47"
Oct 30 05:22:39.655915 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:39.655890    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43eba674-9kf47"]
Oct 30 05:22:39.657940 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:22:39.657915    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4378744_63f2_4b39_a8eb_e32a343b4ade.slice/crio-2c33642ced3dc4d7b57c349fdcbc6139260cb85fb5e437c034568bec77347d50 WatchSource:0}: Error finding container 2c33642ced3dc4d7b57c349fdcbc6139260cb85fb5e437c034568bec77347d50: Status 404 returned error can't find the container with id 2c33642ced3dc4d7b57c349fdcbc6139260cb85fb5e437c034568bec77347d50
Oct 30 05:22:39.696514 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:39.696491    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43eba674-9kf47" event={"ID":"a4378744-63f2-4b39-a8eb-e32a343b4ade","Type":"ContainerStarted","Data":"2c33642ced3dc4d7b57c349fdcbc6139260cb85fb5e437c034568bec77347d50"}
Oct 30 05:22:40.705376 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:40.705344    4874 generic.go:356] "Generic (PLEG): container finished" podID="a4378744-63f2-4b39-a8eb-e32a343b4ade" containerID="3dd24088fe10cb2197d6076e75bb211bd77b5e31151840d07086d2c8599dbb2d" exitCode=0
Oct 30 05:22:40.705752 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:40.705422    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43eba674-9kf47" event={"ID":"a4378744-63f2-4b39-a8eb-e32a343b4ade","Type":"ContainerDied","Data":"3dd24088fe10cb2197d6076e75bb211bd77b5e31151840d07086d2c8599dbb2d"}
Oct 30 05:22:41.860124 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:41.860097    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43eba674-9kf47"
Oct 30 05:22:41.990077 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:41.990048    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4378744-63f2-4b39-a8eb-e32a343b4ade-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4378744-63f2-4b39-a8eb-e32a343b4ade-maas-api-openshift-ca\") pod \"a4378744-63f2-4b39-a8eb-e32a343b4ade\" (UID: \"a4378744-63f2-4b39-a8eb-e32a343b4ade\") "
Oct 30 05:22:41.990276 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:41.990114    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4378744-63f2-4b39-a8eb-e32a343b4ade-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4378744-63f2-4b39-a8eb-e32a343b4ade-maas-api-xks-ca\") pod \"a4378744-63f2-4b39-a8eb-e32a343b4ade\" (UID: \"a4378744-63f2-4b39-a8eb-e32a343b4ade\") "
Oct 30 05:22:41.990276 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:41.990145    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4378744-63f2-4b39-a8eb-e32a343b4ade-kube-api-access-mcznf\" (UniqueName: \"kubernetes.io/projected/a4378744-63f2-4b39-a8eb-e32a343b4ade-kube-api-access-mcznf\") pod \"a4378744-63f2-4b39-a8eb-e32a343b4ade\" (UID: \"a4378744-63f2-4b39-a8eb-e32a343b4ade\") "
Oct 30 05:22:41.990502 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:41.990478    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4378744-63f2-4b39-a8eb-e32a343b4ade-maas-api-openshift-ca" pod "a4378744-63f2-4b39-a8eb-e32a343b4ade" (UID: "a4378744-63f2-4b39-a8eb-e32a343b4ade"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:22:41.992170 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:41.992148    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4378744-63f2-4b39-a8eb-e32a343b4ade-maas-api-xks-ca" pod "a4378744-63f2-4b39-a8eb-e32a343b4ade" (UID: "a4378744-63f2-4b39-a8eb-e32a343b4ade"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:22:41.992283 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:41.992208    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4378744-63f2-4b39-a8eb-e32a343b4ade-kube-api-access-mcznf" pod "a4378744-63f2-4b39-a8eb-e32a343b4ade" (UID: "a4378744-63f2-4b39-a8eb-e32a343b4ade"). InnerVolumeSpecName "kube-api-access-mcznf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:22:42.090979 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:42.090949    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4378744-63f2-4b39-a8eb-e32a343b4ade-maas-api-openshift-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:22:42.090979 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:42.090975    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4378744-63f2-4b39-a8eb-e32a343b4ade-maas-api-xks-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:22:42.091137 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:42.090985    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcznf\" (UniqueName: \"kubernetes.io/projected/a4378744-63f2-4b39-a8eb-e32a343b4ade-kube-api-access-mcznf\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:22:42.716794 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:42.716754    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43eba674-9kf47"
Oct 30 05:22:42.716794 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:42.716785    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43eba674-9kf47" event={"ID":"a4378744-63f2-4b39-a8eb-e32a343b4ade","Type":"ContainerDied","Data":"2c33642ced3dc4d7b57c349fdcbc6139260cb85fb5e437c034568bec77347d50"}
Oct 30 05:22:42.717081 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:42.716871    4874 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c33642ced3dc4d7b57c349fdcbc6139260cb85fb5e437c034568bec77347d50"
Oct 30 05:22:50.433820 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:22:50.433781    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:22:55.608949 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:22:55.608905    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:23:01.455119 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:01.455089    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 05:23:01.461766 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:01.461742    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 05:23:02.445133 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:02.445095    4874 scope.go:122] "RemoveContainer" containerID="cd75a6e9e06218b70e64668d1ed3dfe04ca62b5cc743a009b6fe3e47e4541f43"
Oct 30 05:23:02.453254 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:02.453216    4874 scope.go:122] "RemoveContainer" containerID="60453f38396a3e5a36e7e7dcc8ab6f4043e52bd5389b6bb652f2dd8457e9f919"
Oct 30 05:23:23.508459 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:23.508424    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-58744ff6df-48bdt"]
Oct 30 05:23:23.508941 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:23.508773    4874 cpu_manager.go:414] "containerMap: removing container" podUID="a4378744-63f2-4b39-a8eb-e32a343b4ade" containerName="revoke-keys"
Oct 30 05:23:23.508941 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:23.508784    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4378744-63f2-4b39-a8eb-e32a343b4ade" containerName="revoke-keys"
Oct 30 05:23:23.508941 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:23.508845    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4378744-63f2-4b39-a8eb-e32a343b4ade" containerName="revoke-keys"
Oct 30 05:23:23.511444 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:23.511426    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-58744ff6df-48bdt"
Oct 30 05:23:23.526554 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:23.526536    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-cn9lf\""
Oct 30 05:23:23.537748 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:23.537729    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 05:23:23.540660 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:23.540636    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-58744ff6df-48bdt"]
Oct 30 05:23:23.559842 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:23.559816    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250dbba8-ecd9-409e-ae24-1eadbdb6f7a7-plugins-config-volume\") pod \"payload-pre-processing-58744ff6df-48bdt\" (UID: \"250dbba8-ecd9-409e-ae24-1eadbdb6f7a7\") " pod="openshift-ingress/payload-pre-processing-58744ff6df-48bdt"
Oct 30 05:23:23.559945 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:23.559865    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7vdr\" (UniqueName: \"kubernetes.io/projected/250dbba8-ecd9-409e-ae24-1eadbdb6f7a7-kube-api-access-l7vdr\") pod \"payload-pre-processing-58744ff6df-48bdt\" (UID: \"250dbba8-ecd9-409e-ae24-1eadbdb6f7a7\") " pod="openshift-ingress/payload-pre-processing-58744ff6df-48bdt"
Oct 30 05:23:23.661026 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:23.661001    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250dbba8-ecd9-409e-ae24-1eadbdb6f7a7-plugins-config-volume\") pod \"payload-pre-processing-58744ff6df-48bdt\" (UID: \"250dbba8-ecd9-409e-ae24-1eadbdb6f7a7\") " pod="openshift-ingress/payload-pre-processing-58744ff6df-48bdt"
Oct 30 05:23:23.661132 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:23.661034    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7vdr\" (UniqueName: \"kubernetes.io/projected/250dbba8-ecd9-409e-ae24-1eadbdb6f7a7-kube-api-access-l7vdr\") pod \"payload-pre-processing-58744ff6df-48bdt\" (UID: \"250dbba8-ecd9-409e-ae24-1eadbdb6f7a7\") " pod="openshift-ingress/payload-pre-processing-58744ff6df-48bdt"
Oct 30 05:23:23.661688 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:23.661668    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250dbba8-ecd9-409e-ae24-1eadbdb6f7a7-plugins-config-volume\") pod \"payload-pre-processing-58744ff6df-48bdt\" (UID: \"250dbba8-ecd9-409e-ae24-1eadbdb6f7a7\") " pod="openshift-ingress/payload-pre-processing-58744ff6df-48bdt"
Oct 30 05:23:23.684362 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:23.684335    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7vdr\" (UniqueName: \"kubernetes.io/projected/250dbba8-ecd9-409e-ae24-1eadbdb6f7a7-kube-api-access-l7vdr\") pod \"payload-pre-processing-58744ff6df-48bdt\" (UID: \"250dbba8-ecd9-409e-ae24-1eadbdb6f7a7\") " pod="openshift-ingress/payload-pre-processing-58744ff6df-48bdt"
Oct 30 05:23:23.823319 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:23.823260    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-58744ff6df-48bdt"
Oct 30 05:23:24.125197 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:23:24.125167    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250dbba8_ecd9_409e_ae24_1eadbdb6f7a7.slice/crio-c08d5eaca480983bc67cd645375eed689e2cb21a3eb1b7019cafecbd45775f06 WatchSource:0}: Error finding container c08d5eaca480983bc67cd645375eed689e2cb21a3eb1b7019cafecbd45775f06: Status 404 returned error can't find the container with id c08d5eaca480983bc67cd645375eed689e2cb21a3eb1b7019cafecbd45775f06
Oct 30 05:23:24.125589 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:24.125563    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-58744ff6df-48bdt"]
Oct 30 05:23:24.917484 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:24.917451    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-58744ff6df-48bdt" event={"ID":"250dbba8-ecd9-409e-ae24-1eadbdb6f7a7","Type":"ContainerStarted","Data":"f2f65b377d03bb79fbfe5667f67bf498c5ac5e0f3f9dda765127c5c2f6f8f946"}
Oct 30 05:23:24.917879 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:24.917501    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-58744ff6df-48bdt" event={"ID":"250dbba8-ecd9-409e-ae24-1eadbdb6f7a7","Type":"ContainerStarted","Data":"c08d5eaca480983bc67cd645375eed689e2cb21a3eb1b7019cafecbd45775f06"}
Oct 30 05:23:24.917879 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:24.917564    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-58744ff6df-48bdt"
Oct 30 05:23:24.960466 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:24.960421    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-58744ff6df-48bdt" podStartSLOduration=1.9604087030000001 podStartE2EDuration="1.960408703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:23:24.957777546 +0000 UTC m=+5124.370173119" watchObservedRunningTime="2026-10-30 05:23:24.960408703 +0000 UTC m=+5124.372804277"
Oct 30 05:23:25.608687 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:23:25.608651    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:23:35.924516 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:35.924487    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-58744ff6df-48bdt"
Oct 30 05:23:40.931495 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:23:40.931463    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:23:55.608320 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:23:55.608281    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:24:25.608614 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:24:25.608539    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:24:26.028433 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:24:26.028400    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a00ef9d-s5svp"]
Oct 30 05:24:26.039618 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:24:26.039590    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a00ef9d-s5svp"]
Oct 30 05:24:26.728452 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:24:26.728416    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ebddcf1-9702-4cd7-8931-525ca45c52ed" path="/var/lib/kubelet/pods/6ebddcf1-9702-4cd7-8931-525ca45c52ed/volumes"
Oct 30 05:24:42.028054 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:24:42.028022    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4f5e14b-vp9nb"]
Oct 30 05:24:42.039868 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:24:42.039843    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4f5e14b-vp9nb"]
Oct 30 05:24:42.727390 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:24:42.727359    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a439cc07-946c-4119-bfb8-3f04130f52e8" path="/var/lib/kubelet/pods/a439cc07-946c-4119-bfb8-3f04130f52e8/volumes"
Oct 30 05:24:55.608837 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:24:55.608794    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:25:18.130954 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:25:18.130919    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:25:25.303683 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:25:25.303655    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:25:25.608731 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:25:25.608658    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:25:41.044622 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:25:41.044590    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:25:45.731523 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:25:45.731485    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:25:55.608320 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:25:55.608282    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:26:01.513119 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:26:01.513081    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:26:02.493386 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:26:02.493349    4874 scope.go:122] "RemoveContainer" containerID="6d4ac414f57ca0c7d6c6a1958f6d92b07143ac1cdb9dea9d2d1d17f21f6bba9d"
Oct 30 05:26:02.503728 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:26:02.503694    4874 scope.go:122] "RemoveContainer" containerID="fe26c0e5faf0af386ed72a148b745cd956c06493e4e4227cc08a4e8be103461e"
Oct 30 05:26:17.341184 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:26:17.341140    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:26:25.608282 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:26:25.608216    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:26:41.042585 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:26:41.042553    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:26:51.345352 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:26:51.345320    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:26:55.608800 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:26:55.608760    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:26:56.026778 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:26:56.026740    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:27:01.029004 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:27:01.028969    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lr5lm"]
Oct 30 05:27:01.041276 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:27:01.041228    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lr5lm"]
Oct 30 05:27:02.726803 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:27:02.726768    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="003a071c-7125-4bd6-964a-8e744bab41ae" path="/var/lib/kubelet/pods/003a071c-7125-4bd6-964a-8e744bab41ae/volumes"
Oct 30 05:27:12.941634 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:27:12.941604    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:27:18.548319 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:27:18.547426    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:27:23.009048 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:27:23.009019    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:27:23.027724 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:27:23.027701    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fmrhp"]
Oct 30 05:27:23.039093 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:27:23.039064    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fmrhp"]
Oct 30 05:27:24.727710 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:27:24.727677    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a15cef83-2199-4538-98dd-adda41c623a2" path="/var/lib/kubelet/pods/a15cef83-2199-4538-98dd-adda41c623a2/volumes"
Oct 30 05:27:25.609885 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:27:25.609841    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:27:32.548480 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:27:32.548448    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:27:37.947877 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:27:37.947839    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:27:42.031278 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:27:42.031231    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43eba674-9kf47"]
Oct 30 05:27:42.043384 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:27:42.043352    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43eba674-9kf47"]
Oct 30 05:27:42.727698 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:27:42.727654    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4378744-63f2-4b39-a8eb-e32a343b4ade" path="/var/lib/kubelet/pods/a4378744-63f2-4b39-a8eb-e32a343b4ade/volumes"
Oct 30 05:27:46.140403 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:27:46.140370    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:27:51.430427 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:27:51.430391    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:27:55.608187 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:27:55.608137    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:28:01.484442 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:28:01.484415    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 05:28:01.491207 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:28:01.491186    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 05:28:02.532430 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:28:02.532397    4874 scope.go:122] "RemoveContainer" containerID="3e053b3fa02751ec05a7756804677d67aac3795320d9167c059f07171bd9a818"
Oct 30 05:28:10.540315 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:28:10.540279    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:28:16.426613 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:28:16.426579    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:28:25.608231 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:28:25.608183    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:28:29.042349 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:28:29.042315    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:28:34.349987 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:28:34.349953    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:28:41.102763 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:28:41.102726    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:28:55.609529 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:28:55.609493    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:28:59.100808 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:28:59.100772    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:29:02.551404 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:02.551371    4874 scope.go:122] "RemoveContainer" containerID="3dd24088fe10cb2197d6076e75bb211bd77b5e31151840d07086d2c8599dbb2d"
Oct 30 05:29:02.559759 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:02.559739    4874 scope.go:122] "RemoveContainer" containerID="dd1bd577ba5e84d62b9314b09d9b03791884e43a40139efdf27a32b50b2a486e"
Oct 30 05:29:09.454518 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:09.454483    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:29:25.608343 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:29:25.608307    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:29:26.137657 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:26.137627    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:29:32.738220 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:32.738188    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:29:34.807183 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:34.807150    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch"]
Oct 30 05:29:34.810844 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:34.810820    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch"
Oct 30 05:29:34.836287 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:34.836262    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-042f4c-serving-cert\""
Oct 30 05:29:34.836287 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:34.836281    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 05:29:34.836456 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:34.836321    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gw5bk\""
Oct 30 05:29:34.838037 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:34.838015    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch"]
Oct 30 05:29:34.948888 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:34.948854    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/194505ca-8465-46d7-8a78-49d701b10018-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch\" (UID: \"194505ca-8465-46d7-8a78-49d701b10018\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch"
Oct 30 05:29:34.949069 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:34.948911    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/194505ca-8465-46d7-8a78-49d701b10018-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch\" (UID: \"194505ca-8465-46d7-8a78-49d701b10018\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch"
Oct 30 05:29:34.949069 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:34.949039    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/194505ca-8465-46d7-8a78-49d701b10018-metrics-certs\") pod \"maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch\" (UID: \"194505ca-8465-46d7-8a78-49d701b10018\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch"
Oct 30 05:29:34.949170 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:34.949099    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm9wt\" (UniqueName: \"kubernetes.io/projected/194505ca-8465-46d7-8a78-49d701b10018-kube-api-access-fm9wt\") pod \"maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch\" (UID: \"194505ca-8465-46d7-8a78-49d701b10018\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch"
Oct 30 05:29:35.050084 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.050051    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/194505ca-8465-46d7-8a78-49d701b10018-metrics-certs\") pod \"maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch\" (UID: \"194505ca-8465-46d7-8a78-49d701b10018\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch"
Oct 30 05:29:35.050287 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.050097    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm9wt\" (UniqueName: \"kubernetes.io/projected/194505ca-8465-46d7-8a78-49d701b10018-kube-api-access-fm9wt\") pod \"maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch\" (UID: \"194505ca-8465-46d7-8a78-49d701b10018\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch"
Oct 30 05:29:35.050287 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.050147    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/194505ca-8465-46d7-8a78-49d701b10018-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch\" (UID: \"194505ca-8465-46d7-8a78-49d701b10018\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch"
Oct 30 05:29:35.050287 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.050189    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/194505ca-8465-46d7-8a78-49d701b10018-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch\" (UID: \"194505ca-8465-46d7-8a78-49d701b10018\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch"
Oct 30 05:29:35.050868 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.050845    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/194505ca-8465-46d7-8a78-49d701b10018-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch\" (UID: \"194505ca-8465-46d7-8a78-49d701b10018\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch"
Oct 30 05:29:35.052579 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.052554    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/194505ca-8465-46d7-8a78-49d701b10018-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch\" (UID: \"194505ca-8465-46d7-8a78-49d701b10018\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch"
Oct 30 05:29:35.052729 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.052705    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/194505ca-8465-46d7-8a78-49d701b10018-metrics-certs\") pod \"maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch\" (UID: \"194505ca-8465-46d7-8a78-49d701b10018\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch"
Oct 30 05:29:35.072752 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.072696    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm9wt\" (UniqueName: \"kubernetes.io/projected/194505ca-8465-46d7-8a78-49d701b10018-kube-api-access-fm9wt\") pod \"maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch\" (UID: \"194505ca-8465-46d7-8a78-49d701b10018\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch"
Oct 30 05:29:35.126639 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.126615    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch"
Oct 30 05:29:35.209646 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.209597    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps"]
Oct 30 05:29:35.214176 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.214146    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps"
Oct 30 05:29:35.229904 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.229871    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps"]
Oct 30 05:29:35.231503 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.231461    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-042f4c\""
Oct 30 05:29:35.231712 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.231617    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-d4169098-dockercfg-kjnvt\""
Oct 30 05:29:35.352677 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.352598    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0029d4-522b-4690-9c48-ff130241ab19-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps\" (UID: \"5b0029d4-522b-4690-9c48-ff130241ab19\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps"
Oct 30 05:29:35.352677 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.352639    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knqf6\" (UniqueName: \"kubernetes.io/projected/5b0029d4-522b-4690-9c48-ff130241ab19-kube-api-access-knqf6\") pod \"payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps\" (UID: \"5b0029d4-522b-4690-9c48-ff130241ab19\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps"
Oct 30 05:29:35.453969 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.453942    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0029d4-522b-4690-9c48-ff130241ab19-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps\" (UID: \"5b0029d4-522b-4690-9c48-ff130241ab19\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps"
Oct 30 05:29:35.454122 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.453977    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knqf6\" (UniqueName: \"kubernetes.io/projected/5b0029d4-522b-4690-9c48-ff130241ab19-kube-api-access-knqf6\") pod \"payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps\" (UID: \"5b0029d4-522b-4690-9c48-ff130241ab19\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps"
Oct 30 05:29:35.454644 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.454625    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0029d4-522b-4690-9c48-ff130241ab19-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps\" (UID: \"5b0029d4-522b-4690-9c48-ff130241ab19\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps"
Oct 30 05:29:35.476549 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.476523    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knqf6\" (UniqueName: \"kubernetes.io/projected/5b0029d4-522b-4690-9c48-ff130241ab19-kube-api-access-knqf6\") pod \"payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps\" (UID: \"5b0029d4-522b-4690-9c48-ff130241ab19\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps"
Oct 30 05:29:35.486489 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.486470    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch"]
Oct 30 05:29:35.487781 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:29:35.487756    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod194505ca_8465_46d7_8a78_49d701b10018.slice/crio-50116230457ac28af81e3b37201d222cfec22f10c5658eaba3e15b12d376c2fa WatchSource:0}: Error finding container 50116230457ac28af81e3b37201d222cfec22f10c5658eaba3e15b12d376c2fa: Status 404 returned error can't find the container with id 50116230457ac28af81e3b37201d222cfec22f10c5658eaba3e15b12d376c2fa
Oct 30 05:29:35.489049 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.489030    4874 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 05:29:35.535389 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.535348    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps"
Oct 30 05:29:35.832781 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.832758    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps"]
Oct 30 05:29:35.834852 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:29:35.834825    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0029d4_522b_4690_9c48_ff130241ab19.slice/crio-49c35d3e5613b2a75dcc0c00d8dfa4fc154b5800ad8a8e0a8cabe129e40eae39 WatchSource:0}: Error finding container 49c35d3e5613b2a75dcc0c00d8dfa4fc154b5800ad8a8e0a8cabe129e40eae39: Status 404 returned error can't find the container with id 49c35d3e5613b2a75dcc0c00d8dfa4fc154b5800ad8a8e0a8cabe129e40eae39
Oct 30 05:29:35.917075 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.917041    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch" event={"ID":"194505ca-8465-46d7-8a78-49d701b10018","Type":"ContainerStarted","Data":"50116230457ac28af81e3b37201d222cfec22f10c5658eaba3e15b12d376c2fa"}
Oct 30 05:29:35.918613 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.918590    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps" event={"ID":"5b0029d4-522b-4690-9c48-ff130241ab19","Type":"ContainerStarted","Data":"0216eb44ceac7706110f4559c3671e99c0cdc7a32be4cbf57160d4cb209a5f5d"}
Oct 30 05:29:35.918732 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.918633    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps" event={"ID":"5b0029d4-522b-4690-9c48-ff130241ab19","Type":"ContainerStarted","Data":"49c35d3e5613b2a75dcc0c00d8dfa4fc154b5800ad8a8e0a8cabe129e40eae39"}
Oct 30 05:29:35.918732 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.918709    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps"
Oct 30 05:29:35.958576 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:35.958533    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps" podStartSLOduration=0.958519746 podStartE2EDuration="958.519746ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:29:35.956551334 +0000 UTC m=+5495.368946919" watchObservedRunningTime="2026-10-30 05:29:35.958519746 +0000 UTC m=+5495.370915315"
Oct 30 05:29:36.924119 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:36.924066    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch" event={"ID":"194505ca-8465-46d7-8a78-49d701b10018","Type":"ContainerStarted","Data":"4a660a818082bb16294a65c0258b53c97c14c3caf085850f1611ce1b3d58261a"}
Oct 30 05:29:36.924575 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:36.924214    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch"
Oct 30 05:29:36.965499 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:36.965420    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch" podStartSLOduration=2.52735606 podStartE2EDuration="2.965399962s" totalImagesPullingTime="438.043902ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:29:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:29:36.964554515 +0000 UTC m=+5496.376950087" watchObservedRunningTime="2026-10-30 05:29:36.965399962 +0000 UTC m=+5496.377795533"
Oct 30 05:29:38.334342 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:38.334308    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:29:42.934191 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:42.934164    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch"
Oct 30 05:29:46.927098 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:46.927070    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps"
Oct 30 05:29:53.674081 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:29:53.674044    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:29:55.608556 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:29:55.608519    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:30:00.145192 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:00.145155    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl"]
Oct 30 05:30:00.148597 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:00.148577    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl"
Oct 30 05:30:00.168721 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:00.168690    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl"]
Oct 30 05:30:00.177902 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:00.177882    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:30:00.193074 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:00.193053    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8wrj\" (UniqueName: \"kubernetes.io/projected/f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356-kube-api-access-t8wrj\") pod \"maas-api-key-cleanup-29888970-flgcl\" (UID: \"f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl"
Oct 30 05:30:00.193187 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:00.193116    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888970-flgcl\" (UID: \"f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl"
Oct 30 05:30:00.294201 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:00.294173    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888970-flgcl\" (UID: \"f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl"
Oct 30 05:30:00.294376 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:00.294233    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8wrj\" (UniqueName: \"kubernetes.io/projected/f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356-kube-api-access-t8wrj\") pod \"maas-api-key-cleanup-29888970-flgcl\" (UID: \"f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl"
Oct 30 05:30:00.294829 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:00.294811    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888970-flgcl\" (UID: \"f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl"
Oct 30 05:30:00.316779 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:00.316752    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8wrj\" (UniqueName: \"kubernetes.io/projected/f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356-kube-api-access-t8wrj\") pod \"maas-api-key-cleanup-29888970-flgcl\" (UID: \"f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl"
Oct 30 05:30:00.460917 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:00.460843    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl"
Oct 30 05:30:00.759731 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:00.759695    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl"]
Oct 30 05:30:00.760335 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:30:00.760307    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2be3b16_0253_4c6e_8f3b_ab9fe6dfd356.slice/crio-c21a7471d2761ce6a0dc8efea4c511e1924adc16d3bf981deba531396071a9a9 WatchSource:0}: Error finding container c21a7471d2761ce6a0dc8efea4c511e1924adc16d3bf981deba531396071a9a9: Status 404 returned error can't find the container with id c21a7471d2761ce6a0dc8efea4c511e1924adc16d3bf981deba531396071a9a9
Oct 30 05:30:01.034952 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:01.034867    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl" event={"ID":"f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356","Type":"ContainerStarted","Data":"1b4f34d5145491f72176c1cb33271ddc55b7b01888ddcb85b55aa215c0292997"}
Oct 30 05:30:01.035088 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:01.034955    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl" event={"ID":"f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356","Type":"ContainerStarted","Data":"c21a7471d2761ce6a0dc8efea4c511e1924adc16d3bf981deba531396071a9a9"}
Oct 30 05:30:01.074296 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:01.074216    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl" podStartSLOduration=1.07420157 podStartE2EDuration="1.07420157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:01.073323843 +0000 UTC m=+5520.485719412" watchObservedRunningTime="2026-10-30 05:30:01.07420157 +0000 UTC m=+5520.486597145"
Oct 30 05:30:02.041013 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:02.040983    4874 generic.go:356] "Generic (PLEG): container finished" podID="f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356" containerID="1b4f34d5145491f72176c1cb33271ddc55b7b01888ddcb85b55aa215c0292997" exitCode=60
Oct 30 05:30:02.041013 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:02.041018    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl" event={"ID":"f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356","Type":"ContainerDied","Data":"1b4f34d5145491f72176c1cb33271ddc55b7b01888ddcb85b55aa215c0292997"}
Oct 30 05:30:02.041471 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:02.041290    4874 scope.go:122] "RemoveContainer" containerID="1b4f34d5145491f72176c1cb33271ddc55b7b01888ddcb85b55aa215c0292997"
Oct 30 05:30:03.046878 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:03.046844    4874 generic.go:356] "Generic (PLEG): container finished" podID="f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356" containerID="81909ed32ba2c44f38e7a8aeeb42f7fb5d40b8726387be47026b158afeb8dbe0" exitCode=60
Oct 30 05:30:03.047364 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:03.046919    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl" event={"ID":"f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356","Type":"ContainerDied","Data":"81909ed32ba2c44f38e7a8aeeb42f7fb5d40b8726387be47026b158afeb8dbe0"}
Oct 30 05:30:03.047364 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:03.046982    4874 scope.go:122] "RemoveContainer" containerID="1b4f34d5145491f72176c1cb33271ddc55b7b01888ddcb85b55aa215c0292997"
Oct 30 05:30:03.047364 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:03.047226    4874 scope.go:122] "RemoveContainer" containerID="81909ed32ba2c44f38e7a8aeeb42f7fb5d40b8726387be47026b158afeb8dbe0"
Oct 30 05:30:03.047540 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:30:03.047510    4874 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888970-flgcl_odh-ai-gateway-infra(f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl" podUID="f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356"
Oct 30 05:30:04.053702 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:04.053678    4874 scope.go:122] "RemoveContainer" containerID="81909ed32ba2c44f38e7a8aeeb42f7fb5d40b8726387be47026b158afeb8dbe0"
Oct 30 05:30:04.054082 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:30:04.053895    4874 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888970-flgcl_odh-ai-gateway-infra(f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl" podUID="f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356"
Oct 30 05:30:09.921136 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:09.921097    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:30:14.722222 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:14.722146    4874 scope.go:122] "RemoveContainer" containerID="81909ed32ba2c44f38e7a8aeeb42f7fb5d40b8726387be47026b158afeb8dbe0"
Oct 30 05:30:15.102933 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:15.102897    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl" event={"ID":"f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356","Type":"ContainerStarted","Data":"b690b2bfec79bfa4ede0cd5b3983c06587f611c4d4274ce0d6ec35ea5fe1ac0a"}
Oct 30 05:30:16.108891 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:16.108855    4874 generic.go:356] "Generic (PLEG): container finished" podID="f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356" containerID="b690b2bfec79bfa4ede0cd5b3983c06587f611c4d4274ce0d6ec35ea5fe1ac0a" exitCode=60
Oct 30 05:30:16.109405 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:16.108921    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl" event={"ID":"f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356","Type":"ContainerDied","Data":"b690b2bfec79bfa4ede0cd5b3983c06587f611c4d4274ce0d6ec35ea5fe1ac0a"}
Oct 30 05:30:16.109405 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:16.108981    4874 scope.go:122] "RemoveContainer" containerID="81909ed32ba2c44f38e7a8aeeb42f7fb5d40b8726387be47026b158afeb8dbe0"
Oct 30 05:30:16.109405 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:16.109326    4874 scope.go:122] "RemoveContainer" containerID="b690b2bfec79bfa4ede0cd5b3983c06587f611c4d4274ce0d6ec35ea5fe1ac0a"
Oct 30 05:30:16.109615 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:30:16.109594    4874 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888970-flgcl_odh-ai-gateway-infra(f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl" podUID="f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356"
Oct 30 05:30:16.155432 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:16.155409    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl"]
Oct 30 05:30:17.247010 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:17.246979    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl"
Oct 30 05:30:17.353010 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:17.352977    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356-maas-api-service-ca\") pod \"f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356\" (UID: \"f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356\") "
Oct 30 05:30:17.353170 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:17.353118    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356-kube-api-access-t8wrj\" (UniqueName: \"kubernetes.io/projected/f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356-kube-api-access-t8wrj\") pod \"f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356\" (UID: \"f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356\") "
Oct 30 05:30:17.353412 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:17.353389    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356-maas-api-service-ca" pod "f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356" (UID: "f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:30:17.355278 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:17.355221    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356-kube-api-access-t8wrj" pod "f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356" (UID: "f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356"). InnerVolumeSpecName "kube-api-access-t8wrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:30:17.454154 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:17.454076    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356-maas-api-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:30:17.454154 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:17.454102    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8wrj\" (UniqueName: \"kubernetes.io/projected/f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356-kube-api-access-t8wrj\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:30:18.122770 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:18.122734    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl" event={"ID":"f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356","Type":"ContainerDied","Data":"c21a7471d2761ce6a0dc8efea4c511e1924adc16d3bf981deba531396071a9a9"}
Oct 30 05:30:18.122770 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:18.122767    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl"
Oct 30 05:30:18.123112 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:18.122792    4874 scope.go:122] "RemoveContainer" containerID="b690b2bfec79bfa4ede0cd5b3983c06587f611c4d4274ce0d6ec35ea5fe1ac0a"
Oct 30 05:30:18.173568 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:18.173543    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl"]
Oct 30 05:30:18.184765 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:18.184740    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888970-flgcl"]
Oct 30 05:30:18.727826 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:18.727792    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356" path="/var/lib/kubelet/pods/f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356/volumes"
Oct 30 05:30:25.608732 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:30:25.608694    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:30:34.675631 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:34.675591    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:30:39.211972 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:39.211942    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps"]
Oct 30 05:30:41.008123 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:41.008093    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch"]
Oct 30 05:30:41.008618 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:41.008319    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch" podUID="194505ca-8465-46d7-8a78-49d701b10018" containerName="maas-api" containerID="cri-o://4a660a818082bb16294a65c0258b53c97c14c3caf085850f1611ce1b3d58261a" gracePeriod=30
Oct 30 05:30:41.489485 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:41.489443    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch"
Oct 30 05:30:41.512903 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:41.512868    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/194505ca-8465-46d7-8a78-49d701b10018-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/194505ca-8465-46d7-8a78-49d701b10018-maas-api-tls\") pod \"194505ca-8465-46d7-8a78-49d701b10018\" (UID: \"194505ca-8465-46d7-8a78-49d701b10018\") "
Oct 30 05:30:41.512903 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:41.512903    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/194505ca-8465-46d7-8a78-49d701b10018-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/194505ca-8465-46d7-8a78-49d701b10018-openshift-service-ca\") pod \"194505ca-8465-46d7-8a78-49d701b10018\" (UID: \"194505ca-8465-46d7-8a78-49d701b10018\") "
Oct 30 05:30:41.513129 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:41.512958    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/194505ca-8465-46d7-8a78-49d701b10018-metrics-certs\" (UniqueName: \"kubernetes.io/secret/194505ca-8465-46d7-8a78-49d701b10018-metrics-certs\") pod \"194505ca-8465-46d7-8a78-49d701b10018\" (UID: \"194505ca-8465-46d7-8a78-49d701b10018\") "
Oct 30 05:30:41.513129 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:41.513055    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/194505ca-8465-46d7-8a78-49d701b10018-kube-api-access-fm9wt\" (UniqueName: \"kubernetes.io/projected/194505ca-8465-46d7-8a78-49d701b10018-kube-api-access-fm9wt\") pod \"194505ca-8465-46d7-8a78-49d701b10018\" (UID: \"194505ca-8465-46d7-8a78-49d701b10018\") "
Oct 30 05:30:41.513357 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:41.513325    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/194505ca-8465-46d7-8a78-49d701b10018-openshift-service-ca" pod "194505ca-8465-46d7-8a78-49d701b10018" (UID: "194505ca-8465-46d7-8a78-49d701b10018"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:30:41.515072 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:41.515040    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/194505ca-8465-46d7-8a78-49d701b10018-metrics-certs" pod "194505ca-8465-46d7-8a78-49d701b10018" (UID: "194505ca-8465-46d7-8a78-49d701b10018"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:30:41.515284 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:41.515253    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/194505ca-8465-46d7-8a78-49d701b10018-maas-api-tls" pod "194505ca-8465-46d7-8a78-49d701b10018" (UID: "194505ca-8465-46d7-8a78-49d701b10018"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:30:41.515284 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:41.515263    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194505ca-8465-46d7-8a78-49d701b10018-kube-api-access-fm9wt" pod "194505ca-8465-46d7-8a78-49d701b10018" (UID: "194505ca-8465-46d7-8a78-49d701b10018"). InnerVolumeSpecName "kube-api-access-fm9wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:30:41.614505 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:41.614428    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/194505ca-8465-46d7-8a78-49d701b10018-maas-api-tls\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:30:41.614505 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:41.614458    4874 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/194505ca-8465-46d7-8a78-49d701b10018-openshift-service-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:30:41.614505 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:41.614468    4874 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/194505ca-8465-46d7-8a78-49d701b10018-metrics-certs\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:30:41.614505 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:41.614479    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm9wt\" (UniqueName: \"kubernetes.io/projected/194505ca-8465-46d7-8a78-49d701b10018-kube-api-access-fm9wt\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:30:42.228498 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:42.228458    4874 generic.go:356] "Generic (PLEG): container finished" podID="194505ca-8465-46d7-8a78-49d701b10018" containerID="4a660a818082bb16294a65c0258b53c97c14c3caf085850f1611ce1b3d58261a" exitCode=0
Oct 30 05:30:42.228928 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:42.228503    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch"
Oct 30 05:30:42.228928 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:42.228546    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch" event={"ID":"194505ca-8465-46d7-8a78-49d701b10018","Type":"ContainerDied","Data":"4a660a818082bb16294a65c0258b53c97c14c3caf085850f1611ce1b3d58261a"}
Oct 30 05:30:42.228928 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:42.228592    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch" event={"ID":"194505ca-8465-46d7-8a78-49d701b10018","Type":"ContainerDied","Data":"50116230457ac28af81e3b37201d222cfec22f10c5658eaba3e15b12d376c2fa"}
Oct 30 05:30:42.228928 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:42.228614    4874 scope.go:122] "RemoveContainer" containerID="4a660a818082bb16294a65c0258b53c97c14c3caf085850f1611ce1b3d58261a"
Oct 30 05:30:42.238145 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:42.238127    4874 scope.go:122] "RemoveContainer" containerID="4a660a818082bb16294a65c0258b53c97c14c3caf085850f1611ce1b3d58261a"
Oct 30 05:30:42.238458 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:30:42.238440    4874 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a660a818082bb16294a65c0258b53c97c14c3caf085850f1611ce1b3d58261a\": container with ID starting with 4a660a818082bb16294a65c0258b53c97c14c3caf085850f1611ce1b3d58261a not found: ID does not exist" containerID="4a660a818082bb16294a65c0258b53c97c14c3caf085850f1611ce1b3d58261a"
Oct 30 05:30:42.238513 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:42.238467    4874 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a660a818082bb16294a65c0258b53c97c14c3caf085850f1611ce1b3d58261a"} err="failed to get container status \"4a660a818082bb16294a65c0258b53c97c14c3caf085850f1611ce1b3d58261a\": rpc error: code = NotFound desc = could not find container \"4a660a818082bb16294a65c0258b53c97c14c3caf085850f1611ce1b3d58261a\": container with ID starting with 4a660a818082bb16294a65c0258b53c97c14c3caf085850f1611ce1b3d58261a not found: ID does not exist"
Oct 30 05:30:42.278365 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:42.278342    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch"]
Oct 30 05:30:42.289938 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:42.289919    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-042f4c-78896dd8b9-48zch"]
Oct 30 05:30:42.726712 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:42.726678    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="194505ca-8465-46d7-8a78-49d701b10018" path="/var/lib/kubelet/pods/194505ca-8465-46d7-8a78-49d701b10018/volumes"
Oct 30 05:30:43.439518 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:43.438900    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:30:44.213318 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:44.213218    4874 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps" podUID="5b0029d4-522b-4690-9c48-ff130241ab19" containerName="payload-processing" containerID="cri-o://0216eb44ceac7706110f4559c3671e99c0cdc7a32be4cbf57160d4cb209a5f5d" gracePeriod=25
Oct 30 05:30:46.929879 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:46.929830    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps" podUID="5b0029d4-522b-4690-9c48-ff130241ab19" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:30:49.295490 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.295458    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v9hv7"]
Oct 30 05:30:49.295944 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.295825    4874 cpu_manager.go:414] "containerMap: removing container" podUID="f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356" containerName="cleanup"
Oct 30 05:30:49.295944 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.295837    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356" containerName="cleanup"
Oct 30 05:30:49.295944 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.295850    4874 cpu_manager.go:414] "containerMap: removing container" podUID="194505ca-8465-46d7-8a78-49d701b10018" containerName="maas-api"
Oct 30 05:30:49.295944 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.295857    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="194505ca-8465-46d7-8a78-49d701b10018" containerName="maas-api"
Oct 30 05:30:49.295944 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.295869    4874 cpu_manager.go:414] "containerMap: removing container" podUID="f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356" containerName="cleanup"
Oct 30 05:30:49.295944 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.295874    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356" containerName="cleanup"
Oct 30 05:30:49.295944 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.295889    4874 cpu_manager.go:414] "containerMap: removing container" podUID="f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356" containerName="cleanup"
Oct 30 05:30:49.295944 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.295896    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356" containerName="cleanup"
Oct 30 05:30:49.296232 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.295955    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356" containerName="cleanup"
Oct 30 05:30:49.296232 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.295965    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="194505ca-8465-46d7-8a78-49d701b10018" containerName="maas-api"
Oct 30 05:30:49.296232 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.295976    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356" containerName="cleanup"
Oct 30 05:30:49.296232 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.295984    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2be3b16-0253-4c6e-8f3b-ab9fe6dfd356" containerName="cleanup"
Oct 30 05:30:49.300110 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.300093    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v9hv7"
Oct 30 05:30:49.318066 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.318040    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v9hv7"]
Oct 30 05:30:49.375184 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.375157    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 05:30:49.375335 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.375258    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\""
Oct 30 05:30:49.384637 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.384614    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/673d44c2-a7c3-46cf-bf40-857c6adbdf4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-v9hv7\" (UID: \"673d44c2-a7c3-46cf-bf40-857c6adbdf4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v9hv7"
Oct 30 05:30:49.384735 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.384662    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/673d44c2-a7c3-46cf-bf40-857c6adbdf4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-v9hv7\" (UID: \"673d44c2-a7c3-46cf-bf40-857c6adbdf4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v9hv7"
Oct 30 05:30:49.384779 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.384765    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjjxp\" (UniqueName: \"kubernetes.io/projected/673d44c2-a7c3-46cf-bf40-857c6adbdf4f-kube-api-access-kjjxp\") pod \"maas-api-revoke-sub-3d4156f1-v9hv7\" (UID: \"673d44c2-a7c3-46cf-bf40-857c6adbdf4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v9hv7"
Oct 30 05:30:49.485186 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.485157    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/673d44c2-a7c3-46cf-bf40-857c6adbdf4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-v9hv7\" (UID: \"673d44c2-a7c3-46cf-bf40-857c6adbdf4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v9hv7"
Oct 30 05:30:49.485320 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.485209    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjjxp\" (UniqueName: \"kubernetes.io/projected/673d44c2-a7c3-46cf-bf40-857c6adbdf4f-kube-api-access-kjjxp\") pod \"maas-api-revoke-sub-3d4156f1-v9hv7\" (UID: \"673d44c2-a7c3-46cf-bf40-857c6adbdf4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v9hv7"
Oct 30 05:30:49.485320 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.485267    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/673d44c2-a7c3-46cf-bf40-857c6adbdf4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-v9hv7\" (UID: \"673d44c2-a7c3-46cf-bf40-857c6adbdf4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v9hv7"
Oct 30 05:30:49.485834 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.485803    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/673d44c2-a7c3-46cf-bf40-857c6adbdf4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-v9hv7\" (UID: \"673d44c2-a7c3-46cf-bf40-857c6adbdf4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v9hv7"
Oct 30 05:30:49.487622 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.487595    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/673d44c2-a7c3-46cf-bf40-857c6adbdf4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-v9hv7\" (UID: \"673d44c2-a7c3-46cf-bf40-857c6adbdf4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v9hv7"
Oct 30 05:30:49.508895 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.508873    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjjxp\" (UniqueName: \"kubernetes.io/projected/673d44c2-a7c3-46cf-bf40-857c6adbdf4f-kube-api-access-kjjxp\") pod \"maas-api-revoke-sub-3d4156f1-v9hv7\" (UID: \"673d44c2-a7c3-46cf-bf40-857c6adbdf4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v9hv7"
Oct 30 05:30:49.612620 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.612552    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v9hv7"
Oct 30 05:30:49.911737 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:49.911705    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v9hv7"]
Oct 30 05:30:49.912989 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:30:49.912962    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod673d44c2_a7c3_46cf_bf40_857c6adbdf4f.slice/crio-4261bec0051b5aa27a1c36a3a83112309cce0ebdf6f77d0eb6ae604bf3dff2cb WatchSource:0}: Error finding container 4261bec0051b5aa27a1c36a3a83112309cce0ebdf6f77d0eb6ae604bf3dff2cb: Status 404 returned error can't find the container with id 4261bec0051b5aa27a1c36a3a83112309cce0ebdf6f77d0eb6ae604bf3dff2cb
Oct 30 05:30:50.268738 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:50.268704    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v9hv7" event={"ID":"673d44c2-a7c3-46cf-bf40-857c6adbdf4f","Type":"ContainerStarted","Data":"9b20f83a8b2e55c8352a5608907c10bf2f2318b901237285e904760dc645ec35"}
Oct 30 05:30:50.268937 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:50.268764    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v9hv7" event={"ID":"673d44c2-a7c3-46cf-bf40-857c6adbdf4f","Type":"ContainerStarted","Data":"4261bec0051b5aa27a1c36a3a83112309cce0ebdf6f77d0eb6ae604bf3dff2cb"}
Oct 30 05:30:50.309424 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:50.309376    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v9hv7" podStartSLOduration=1.3093644150000001 podStartE2EDuration="1.309364415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:50.308695045 +0000 UTC m=+5569.721090618" watchObservedRunningTime="2026-10-30 05:30:50.309364415 +0000 UTC m=+5569.721759987"
Oct 30 05:30:51.275214 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:51.275176    4874 generic.go:356] "Generic (PLEG): container finished" podID="673d44c2-a7c3-46cf-bf40-857c6adbdf4f" containerID="9b20f83a8b2e55c8352a5608907c10bf2f2318b901237285e904760dc645ec35" exitCode=0
Oct 30 05:30:51.275441 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:51.275266    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v9hv7" event={"ID":"673d44c2-a7c3-46cf-bf40-857c6adbdf4f","Type":"ContainerDied","Data":"9b20f83a8b2e55c8352a5608907c10bf2f2318b901237285e904760dc645ec35"}
Oct 30 05:30:52.407507 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:52.407477    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v9hv7"
Oct 30 05:30:52.514020 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:52.513990    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/673d44c2-a7c3-46cf-bf40-857c6adbdf4f-kube-api-access-kjjxp\" (UniqueName: \"kubernetes.io/projected/673d44c2-a7c3-46cf-bf40-857c6adbdf4f-kube-api-access-kjjxp\") pod \"673d44c2-a7c3-46cf-bf40-857c6adbdf4f\" (UID: \"673d44c2-a7c3-46cf-bf40-857c6adbdf4f\") "
Oct 30 05:30:52.514193 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:52.514040    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/673d44c2-a7c3-46cf-bf40-857c6adbdf4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/673d44c2-a7c3-46cf-bf40-857c6adbdf4f-maas-api-xks-ca\") pod \"673d44c2-a7c3-46cf-bf40-857c6adbdf4f\" (UID: \"673d44c2-a7c3-46cf-bf40-857c6adbdf4f\") "
Oct 30 05:30:52.514193 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:52.514063    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/673d44c2-a7c3-46cf-bf40-857c6adbdf4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/673d44c2-a7c3-46cf-bf40-857c6adbdf4f-maas-api-openshift-ca\") pod \"673d44c2-a7c3-46cf-bf40-857c6adbdf4f\" (UID: \"673d44c2-a7c3-46cf-bf40-857c6adbdf4f\") "
Oct 30 05:30:52.514471 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:52.514445    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/673d44c2-a7c3-46cf-bf40-857c6adbdf4f-maas-api-openshift-ca" pod "673d44c2-a7c3-46cf-bf40-857c6adbdf4f" (UID: "673d44c2-a7c3-46cf-bf40-857c6adbdf4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:30:52.516110 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:52.516082    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/673d44c2-a7c3-46cf-bf40-857c6adbdf4f-maas-api-xks-ca" pod "673d44c2-a7c3-46cf-bf40-857c6adbdf4f" (UID: "673d44c2-a7c3-46cf-bf40-857c6adbdf4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 05:30:52.516198 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:52.516141    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/673d44c2-a7c3-46cf-bf40-857c6adbdf4f-kube-api-access-kjjxp" pod "673d44c2-a7c3-46cf-bf40-857c6adbdf4f" (UID: "673d44c2-a7c3-46cf-bf40-857c6adbdf4f"). InnerVolumeSpecName "kube-api-access-kjjxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:30:52.615375 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:52.615294    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjjxp\" (UniqueName: \"kubernetes.io/projected/673d44c2-a7c3-46cf-bf40-857c6adbdf4f-kube-api-access-kjjxp\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:30:52.615375 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:52.615331    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/673d44c2-a7c3-46cf-bf40-857c6adbdf4f-maas-api-xks-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:30:52.615375 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:52.615346    4874 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/673d44c2-a7c3-46cf-bf40-857c6adbdf4f-maas-api-openshift-ca\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:30:53.284424 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:53.284395    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v9hv7"
Oct 30 05:30:53.284624 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:53.284434    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v9hv7" event={"ID":"673d44c2-a7c3-46cf-bf40-857c6adbdf4f","Type":"ContainerDied","Data":"4261bec0051b5aa27a1c36a3a83112309cce0ebdf6f77d0eb6ae604bf3dff2cb"}
Oct 30 05:30:53.284624 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:53.284489    4874 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4261bec0051b5aa27a1c36a3a83112309cce0ebdf6f77d0eb6ae604bf3dff2cb"
Oct 30 05:30:55.608745 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:30:55.608710    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:30:56.925544 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:30:56.925502    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps" podUID="5b0029d4-522b-4690-9c48-ff130241ab19" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 05:31:01.525464 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:31:01.525426    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:31:04.356896 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:31:04.356860    4874 generic.go:356] "Generic (PLEG): container finished" podID="5b0029d4-522b-4690-9c48-ff130241ab19" containerID="0216eb44ceac7706110f4559c3671e99c0cdc7a32be4cbf57160d4cb209a5f5d" exitCode=0
Oct 30 05:31:04.357296 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:31:04.356990    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps" event={"ID":"5b0029d4-522b-4690-9c48-ff130241ab19","Type":"ContainerDied","Data":"0216eb44ceac7706110f4559c3671e99c0cdc7a32be4cbf57160d4cb209a5f5d"}
Oct 30 05:31:04.452524 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:31:04.452501    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps"
Oct 30 05:31:04.577097 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:31:04.577064    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b0029d4-522b-4690-9c48-ff130241ab19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0029d4-522b-4690-9c48-ff130241ab19-plugins-config-volume\") pod \"5b0029d4-522b-4690-9c48-ff130241ab19\" (UID: \"5b0029d4-522b-4690-9c48-ff130241ab19\") "
Oct 30 05:31:04.577300 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:31:04.577116    4874 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b0029d4-522b-4690-9c48-ff130241ab19-kube-api-access-knqf6\" (UniqueName: \"kubernetes.io/projected/5b0029d4-522b-4690-9c48-ff130241ab19-kube-api-access-knqf6\") pod \"5b0029d4-522b-4690-9c48-ff130241ab19\" (UID: \"5b0029d4-522b-4690-9c48-ff130241ab19\") "
Oct 30 05:31:04.577553 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:31:04.577528    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0029d4-522b-4690-9c48-ff130241ab19-plugins-config-volume" pod "5b0029d4-522b-4690-9c48-ff130241ab19" (UID: "5b0029d4-522b-4690-9c48-ff130241ab19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 05:31:04.579214 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:31:04.579194    4874 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0029d4-522b-4690-9c48-ff130241ab19-kube-api-access-knqf6" pod "5b0029d4-522b-4690-9c48-ff130241ab19" (UID: "5b0029d4-522b-4690-9c48-ff130241ab19"). InnerVolumeSpecName "kube-api-access-knqf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 05:31:04.678257 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:31:04.678221    4874 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0029d4-522b-4690-9c48-ff130241ab19-plugins-config-volume\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:31:04.678408 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:31:04.678271    4874 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knqf6\" (UniqueName: \"kubernetes.io/projected/5b0029d4-522b-4690-9c48-ff130241ab19-kube-api-access-knqf6\") on node \"ip-10-0-138-160.ec2.internal\" DevicePath \"\""
Oct 30 05:31:05.362574 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:31:05.362543    4874 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps"
Oct 30 05:31:05.363058 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:31:05.362580    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps" event={"ID":"5b0029d4-522b-4690-9c48-ff130241ab19","Type":"ContainerDied","Data":"49c35d3e5613b2a75dcc0c00d8dfa4fc154b5800ad8a8e0a8cabe129e40eae39"}
Oct 30 05:31:05.363058 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:31:05.362643    4874 scope.go:122] "RemoveContainer" containerID="0216eb44ceac7706110f4559c3671e99c0cdc7a32be4cbf57160d4cb209a5f5d"
Oct 30 05:31:05.415969 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:31:05.415934    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps"]
Oct 30 05:31:05.427114 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:31:05.427086    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-042f4c-6b668c9895n9ps"]
Oct 30 05:31:06.727392 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:31:06.727357    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b0029d4-522b-4690-9c48-ff130241ab19" path="/var/lib/kubelet/pods/5b0029d4-522b-4690-9c48-ff130241ab19/volumes"
Oct 30 05:31:25.608386 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:31:25.608352    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:31:55.608066 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:31:55.608019    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:32:25.608005 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:32:25.607970    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:32:31.151262 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:32:31.151199    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:32:46.355046 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:32:46.355013    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:32:51.946917 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:32:51.946878    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:32:55.608382 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:32:55.608345    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:32:56.331138 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:32:56.331105    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:33:01.511708 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:01.511677    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 05:33:01.518935 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:01.518915    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 05:33:02.167945 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.167910    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7b94bfd958-hbpx2"]
Oct 30 05:33:02.168318 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.168304    4874 cpu_manager.go:414] "containerMap: removing container" podUID="673d44c2-a7c3-46cf-bf40-857c6adbdf4f" containerName="revoke-keys"
Oct 30 05:33:02.168381 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.168320    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="673d44c2-a7c3-46cf-bf40-857c6adbdf4f" containerName="revoke-keys"
Oct 30 05:33:02.168381 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.168343    4874 cpu_manager.go:414] "containerMap: removing container" podUID="5b0029d4-522b-4690-9c48-ff130241ab19" containerName="payload-processing"
Oct 30 05:33:02.168381 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.168349    4874 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b0029d4-522b-4690-9c48-ff130241ab19" containerName="payload-processing"
Oct 30 05:33:02.168479 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.168402    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="673d44c2-a7c3-46cf-bf40-857c6adbdf4f" containerName="revoke-keys"
Oct 30 05:33:02.168479 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.168416    4874 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b0029d4-522b-4690-9c48-ff130241ab19" containerName="payload-processing"
Oct 30 05:33:02.171353 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.171333    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b94bfd958-hbpx2"
Oct 30 05:33:02.198372 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.198344    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b94bfd958-hbpx2"]
Oct 30 05:33:02.275473 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.275438    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gw5bk\""
Oct 30 05:33:02.275473 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.275444    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 05:33:02.275677 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.275497    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 05:33:02.290863 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.290834    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bf464f9-3926-440b-9be7-1e4ed79ff276-openshift-service-ca\") pod \"maas-api-7b94bfd958-hbpx2\" (UID: \"8bf464f9-3926-440b-9be7-1e4ed79ff276\") " pod="odh-ai-gateway-infra/maas-api-7b94bfd958-hbpx2"
Oct 30 05:33:02.290980 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.290871    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bf464f9-3926-440b-9be7-1e4ed79ff276-maas-api-tls\") pod \"maas-api-7b94bfd958-hbpx2\" (UID: \"8bf464f9-3926-440b-9be7-1e4ed79ff276\") " pod="odh-ai-gateway-infra/maas-api-7b94bfd958-hbpx2"
Oct 30 05:33:02.290980 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.290944    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf464f9-3926-440b-9be7-1e4ed79ff276-metrics-certs\") pod \"maas-api-7b94bfd958-hbpx2\" (UID: \"8bf464f9-3926-440b-9be7-1e4ed79ff276\") " pod="odh-ai-gateway-infra/maas-api-7b94bfd958-hbpx2"
Oct 30 05:33:02.290980 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.290975    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n662m\" (UniqueName: \"kubernetes.io/projected/8bf464f9-3926-440b-9be7-1e4ed79ff276-kube-api-access-n662m\") pod \"maas-api-7b94bfd958-hbpx2\" (UID: \"8bf464f9-3926-440b-9be7-1e4ed79ff276\") " pod="odh-ai-gateway-infra/maas-api-7b94bfd958-hbpx2"
Oct 30 05:33:02.392089 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.392065    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bf464f9-3926-440b-9be7-1e4ed79ff276-openshift-service-ca\") pod \"maas-api-7b94bfd958-hbpx2\" (UID: \"8bf464f9-3926-440b-9be7-1e4ed79ff276\") " pod="odh-ai-gateway-infra/maas-api-7b94bfd958-hbpx2"
Oct 30 05:33:02.392212 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.392100    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bf464f9-3926-440b-9be7-1e4ed79ff276-maas-api-tls\") pod \"maas-api-7b94bfd958-hbpx2\" (UID: \"8bf464f9-3926-440b-9be7-1e4ed79ff276\") " pod="odh-ai-gateway-infra/maas-api-7b94bfd958-hbpx2"
Oct 30 05:33:02.392212 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.392126    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf464f9-3926-440b-9be7-1e4ed79ff276-metrics-certs\") pod \"maas-api-7b94bfd958-hbpx2\" (UID: \"8bf464f9-3926-440b-9be7-1e4ed79ff276\") " pod="odh-ai-gateway-infra/maas-api-7b94bfd958-hbpx2"
Oct 30 05:33:02.392212 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.392155    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n662m\" (UniqueName: \"kubernetes.io/projected/8bf464f9-3926-440b-9be7-1e4ed79ff276-kube-api-access-n662m\") pod \"maas-api-7b94bfd958-hbpx2\" (UID: \"8bf464f9-3926-440b-9be7-1e4ed79ff276\") " pod="odh-ai-gateway-infra/maas-api-7b94bfd958-hbpx2"
Oct 30 05:33:02.392861 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.392842    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bf464f9-3926-440b-9be7-1e4ed79ff276-openshift-service-ca\") pod \"maas-api-7b94bfd958-hbpx2\" (UID: \"8bf464f9-3926-440b-9be7-1e4ed79ff276\") " pod="odh-ai-gateway-infra/maas-api-7b94bfd958-hbpx2"
Oct 30 05:33:02.394556 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.394533    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bf464f9-3926-440b-9be7-1e4ed79ff276-maas-api-tls\") pod \"maas-api-7b94bfd958-hbpx2\" (UID: \"8bf464f9-3926-440b-9be7-1e4ed79ff276\") " pod="odh-ai-gateway-infra/maas-api-7b94bfd958-hbpx2"
Oct 30 05:33:02.394658 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.394617    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf464f9-3926-440b-9be7-1e4ed79ff276-metrics-certs\") pod \"maas-api-7b94bfd958-hbpx2\" (UID: \"8bf464f9-3926-440b-9be7-1e4ed79ff276\") " pod="odh-ai-gateway-infra/maas-api-7b94bfd958-hbpx2"
Oct 30 05:33:02.414316 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.414296    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n662m\" (UniqueName: \"kubernetes.io/projected/8bf464f9-3926-440b-9be7-1e4ed79ff276-kube-api-access-n662m\") pod \"maas-api-7b94bfd958-hbpx2\" (UID: \"8bf464f9-3926-440b-9be7-1e4ed79ff276\") " pod="odh-ai-gateway-infra/maas-api-7b94bfd958-hbpx2"
Oct 30 05:33:02.488259 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.488217    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b94bfd958-hbpx2"
Oct 30 05:33:02.796391 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.796367    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b94bfd958-hbpx2"]
Oct 30 05:33:02.797626 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:33:02.797602    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf464f9_3926_440b_9be7_1e4ed79ff276.slice/crio-115ea949ba7e0765f8fd5ba4fe01a48385ea5dc93c186c57e0ab98873f921907 WatchSource:0}: Error finding container 115ea949ba7e0765f8fd5ba4fe01a48385ea5dc93c186c57e0ab98873f921907: Status 404 returned error can't find the container with id 115ea949ba7e0765f8fd5ba4fe01a48385ea5dc93c186c57e0ab98873f921907
Oct 30 05:33:02.934763 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:02.934730    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b94bfd958-hbpx2" event={"ID":"8bf464f9-3926-440b-9be7-1e4ed79ff276","Type":"ContainerStarted","Data":"115ea949ba7e0765f8fd5ba4fe01a48385ea5dc93c186c57e0ab98873f921907"}
Oct 30 05:33:03.940932 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:03.940885    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b94bfd958-hbpx2" event={"ID":"8bf464f9-3926-440b-9be7-1e4ed79ff276","Type":"ContainerStarted","Data":"d1506ce005f5b00c389a08b8d59fd37ea5af2172b3f48ec153e78998851192fc"}
Oct 30 05:33:03.941373 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:03.941033    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7b94bfd958-hbpx2"
Oct 30 05:33:03.995146 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:03.995098    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7b94bfd958-hbpx2" podStartSLOduration=1.609240655 podStartE2EDuration="1.995087575s" totalImagesPullingTime="385.84692ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:33:03.994656794 +0000 UTC m=+5703.407052367" watchObservedRunningTime="2026-10-30 05:33:03.995087575 +0000 UTC m=+5703.407483203"
Oct 30 05:33:09.950159 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:09.950122    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7b94bfd958-hbpx2"
Oct 30 05:33:13.215631 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:13.215594    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:33:25.608888 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:33:25.608845    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:33:43.029567 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:43.029537    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:33:47.917728 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:33:47.917696    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:33:55.608624 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:33:55.608582    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:34:04.907294 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:34:04.907264    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:34:09.644415 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:34:09.644378    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:34:15.763842 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:34:15.763790    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:34:25.609129 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:34:25.609092    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:34:35.915654 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:34:35.915621    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:34:46.249538 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:34:46.249503    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:34:51.132692 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:34:51.132662    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:34:55.752573 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:34:55.752512    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:34:56.732221 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:34:56.732185    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hz6cz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:34:56.732521 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:34:56.732491    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz" podUID="805efc98-2476-40e3-9613-69336ac75dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:34:59.225580 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:34:59.225544    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:35:17.635164 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:35:17.635133    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:35:25.608325 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:35:25.608288    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:35:27.958107 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:35:27.958075    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:35:33.376282 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:35:33.375095    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:35:44.749200 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:35:44.749172    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:35:53.029129 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:35:53.029099    4874 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v9hv7"]
Oct 30 05:35:53.040280 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:35:53.040255    4874 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v9hv7"]
Oct 30 05:35:54.726981 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:35:54.726945    4874 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="673d44c2-a7c3-46cf-bf40-857c6adbdf4f" path="/var/lib/kubelet/pods/673d44c2-a7c3-46cf-bf40-857c6adbdf4f/volumes"
Oct 30 05:35:55.608988 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:35:55.608956    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:36:01.697655 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:36:01.697619    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:36:25.608621 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:36:25.608537    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:36:55.736263 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:36:55.734280    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:37:02.691956 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:37:02.691923    4874 scope.go:122] "RemoveContainer" containerID="9b20f83a8b2e55c8352a5608907c10bf2f2318b901237285e904760dc645ec35"
Oct 30 05:37:06.060264 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:37:06.059335    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hwcsp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:37:06.060264 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:37:06.059461    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hwcsp" podUID="62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:37:25.781524 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:37:25.781482    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:37:26.726471 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:37:26.726424    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hz6cz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:37:26.727300 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:37:26.727262    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz" podUID="805efc98-2476-40e3-9613-69336ac75dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:37:55.111177 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:37:55.111146    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg5td"]
Oct 30 05:37:55.711751 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:37:55.711711    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:37:56.744399 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:37:56.744363    4874 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hz6cz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 05:37:56.745035 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:37:56.745001    4874 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz6cz" podUID="805efc98-2476-40e3-9613-69336ac75dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 05:38:01.545085 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:01.545041    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 05:38:01.557549 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:01.557524    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-244hd_97de5e90-d514-4b8c-8917-2e9894978aaa/ovn-acl-logging/0.log"
Oct 30 05:38:04.357193 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:04.357161    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-757f4bf64c-szwjz_36ea66f0-e11f-4da2-a421-1498c0287776/authorino/0.log"
Oct 30 05:38:17.256795 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:17.256757    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f4559f84c-tdxj7_24d61891-9970-444b-9a0d-80a7e911c420/manager/0.log"
Oct 30 05:38:18.822395 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:18.822364    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7b94bfd958-hbpx2_8bf464f9-3926-440b-9be7-1e4ed79ff276/maas-api/0.log"
Oct 30 05:38:22.835519 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:22.835487    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-22p96_5acd9635-8003-4366-b5ed-a5ad9641d38e/postgres/0.log"
Oct 30 05:38:24.729289 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:24.729257    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-757f4bf64c-szwjz_36ea66f0-e11f-4da2-a421-1498c0287776/authorino/0.log"
Oct 30 05:38:25.332782 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:25.332749    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vmffb_93fe6d14-eca6-492a-9237-75362698993f/kuadrant-console-plugin/0.log"
Oct 30 05:38:25.608560 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:38:25.608485    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:38:25.822015 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:25.821982    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-vg5td_b90621d3-240a-489c-9ed2-6c1913a13972/limitador/0.log"
Oct 30 05:38:27.162339 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:27.162308    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-7pv6t_6e6ab204-9f50-4bd2-8b0e-0044278d820d/discovery/0.log"
Oct 30 05:38:27.562609 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:27.562581    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-685cdf9dcb-sgtk5_38243efb-429f-450b-a62a-6b55c62330a6/kube-auth-proxy/0.log"
Oct 30 05:38:27.976095 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:27.976059    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-fh868_7059d28d-3eca-44fa-80c4-ac634a59aaf7/istio-proxy/0.log"
Oct 30 05:38:28.610409 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:28.610375    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-hwcsp_62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69/istio-proxy/0.log"
Oct 30 05:38:28.816939 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:28.816908    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-hz6cz_805efc98-2476-40e3-9613-69336ac75dca/istio-proxy/0.log"
Oct 30 05:38:29.645460 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:29.645433    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-x8mpd_c2ea846a-5ce0-424a-bb80-ec759b592c37/istio-proxy/0.log"
Oct 30 05:38:30.024789 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:30.024751    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-58744ff6df-48bdt_250dbba8-ecd9-409e-ae24-1eadbdb6f7a7/payload-pre-processing/0.log"
Oct 30 05:38:31.749168 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:31.749139    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-l8nmh_d775e3cc-9ee3-4b5c-a155-e26d1a854eab/main/0.log"
Oct 30 05:38:32.157578 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:32.157494    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8jr5sp_522d3048-9568-4813-b2ad-69f2ea7defb8/main/0.log"
Oct 30 05:38:32.352873 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:32.352841    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-n6b8k_c2f77027-4d74-47c0-8593-560a26c071e9/main/0.log"
Oct 30 05:38:43.551682 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:43.551647    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9xfkr/must-gather-ftcpv"]
Oct 30 05:38:43.554342 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:43.554324    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xfkr/must-gather-ftcpv"
Oct 30 05:38:43.574419 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:43.574391    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xfkr/must-gather-ftcpv"]
Oct 30 05:38:43.582733 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:43.582713    4874 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9xfkr\"/\"default-dockercfg-x74pv\""
Oct 30 05:38:43.584056 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:43.584040    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9xfkr\"/\"openshift-service-ca.crt\""
Oct 30 05:38:43.584144 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:43.584126    4874 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9xfkr\"/\"kube-root-ca.crt\""
Oct 30 05:38:43.643882 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:43.643853    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6xgl\" (UniqueName: \"kubernetes.io/projected/b99406bf-2463-4f6a-afac-35ed5289f343-kube-api-access-v6xgl\") pod \"must-gather-ftcpv\" (UID: \"b99406bf-2463-4f6a-afac-35ed5289f343\") " pod="openshift-must-gather-9xfkr/must-gather-ftcpv"
Oct 30 05:38:43.644016 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:43.643907    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b99406bf-2463-4f6a-afac-35ed5289f343-must-gather-output\") pod \"must-gather-ftcpv\" (UID: \"b99406bf-2463-4f6a-afac-35ed5289f343\") " pod="openshift-must-gather-9xfkr/must-gather-ftcpv"
Oct 30 05:38:43.745115 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:43.745081    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6xgl\" (UniqueName: \"kubernetes.io/projected/b99406bf-2463-4f6a-afac-35ed5289f343-kube-api-access-v6xgl\") pod \"must-gather-ftcpv\" (UID: \"b99406bf-2463-4f6a-afac-35ed5289f343\") " pod="openshift-must-gather-9xfkr/must-gather-ftcpv"
Oct 30 05:38:43.745269 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:43.745132    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b99406bf-2463-4f6a-afac-35ed5289f343-must-gather-output\") pod \"must-gather-ftcpv\" (UID: \"b99406bf-2463-4f6a-afac-35ed5289f343\") " pod="openshift-must-gather-9xfkr/must-gather-ftcpv"
Oct 30 05:38:43.745486 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:43.745469    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b99406bf-2463-4f6a-afac-35ed5289f343-must-gather-output\") pod \"must-gather-ftcpv\" (UID: \"b99406bf-2463-4f6a-afac-35ed5289f343\") " pod="openshift-must-gather-9xfkr/must-gather-ftcpv"
Oct 30 05:38:43.767975 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:43.767953    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6xgl\" (UniqueName: \"kubernetes.io/projected/b99406bf-2463-4f6a-afac-35ed5289f343-kube-api-access-v6xgl\") pod \"must-gather-ftcpv\" (UID: \"b99406bf-2463-4f6a-afac-35ed5289f343\") " pod="openshift-must-gather-9xfkr/must-gather-ftcpv"
Oct 30 05:38:43.865751 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:43.865683    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xfkr/must-gather-ftcpv"
Oct 30 05:38:44.161275 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:44.161250    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xfkr/must-gather-ftcpv"]
Oct 30 05:38:44.162795 ip-10-0-138-160 kubenswrapper[4874]: W1030 05:38:44.162768    4874 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb99406bf_2463_4f6a_afac_35ed5289f343.slice/crio-0c1e1a88bc10cb1c47b265388f1fcd497273fa98653f9c58a3d2fd9f18749cd5 WatchSource:0}: Error finding container 0c1e1a88bc10cb1c47b265388f1fcd497273fa98653f9c58a3d2fd9f18749cd5: Status 404 returned error can't find the container with id 0c1e1a88bc10cb1c47b265388f1fcd497273fa98653f9c58a3d2fd9f18749cd5
Oct 30 05:38:44.164579 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:44.164562    4874 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 05:38:45.083168 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:45.083135    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xfkr/must-gather-ftcpv" event={"ID":"b99406bf-2463-4f6a-afac-35ed5289f343","Type":"ContainerStarted","Data":"fb5d68d9edebc07bca6c2fc1dad524b5af4b38f217123612c3f0c2be7a59f49b"}
Oct 30 05:38:45.083512 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:45.083178    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xfkr/must-gather-ftcpv" event={"ID":"b99406bf-2463-4f6a-afac-35ed5289f343","Type":"ContainerStarted","Data":"0c1e1a88bc10cb1c47b265388f1fcd497273fa98653f9c58a3d2fd9f18749cd5"}
Oct 30 05:38:46.089829 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:46.089795    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xfkr/must-gather-ftcpv" event={"ID":"b99406bf-2463-4f6a-afac-35ed5289f343","Type":"ContainerStarted","Data":"3524ea3eb6162330c08116482211253a15f27766defc2e6b135edf539ab75476"}
Oct 30 05:38:46.129257 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:46.129183    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9xfkr/must-gather-ftcpv" podStartSLOduration=2.398050397 podStartE2EDuration="3.12916765s" totalImagesPullingTime="731.117253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:38:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:38:46.127580621 +0000 UTC m=+6045.539976193" watchObservedRunningTime="2026-10-30 05:38:46.12916765 +0000 UTC m=+6045.541563252"
Oct 30 05:38:47.731821 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:47.731790    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cbfm8_bad69b26-9de8-4c2b-b19c-8a96271f5c86/global-pull-secret-syncer/0.log"
Oct 30 05:38:49.096260 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:49.095778    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vnv9w_e385d23f-6371-4f69-97b1-6a10303c048f/konnectivity-agent/0.log"
Oct 30 05:38:49.219259 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:49.213900    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-160.ec2.internal_bc4b060adf5b640eabc59287108542f2/haproxy/0.log"
Oct 30 05:38:52.063012 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:52.050657    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f4559f84c-tdxj7_24d61891-9970-444b-9a0d-80a7e911c420/manager/0.log"
Oct 30 05:38:52.704257 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:52.685485    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f4559f84c-tdxj7_24d61891-9970-444b-9a0d-80a7e911c420/manager/0.log"
Oct 30 05:38:55.635918 ip-10-0-138-160 kubenswrapper[4874]: E1030 05:38:55.635888    4874 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8qhbh" containerName="extractor"
Oct 30 05:38:57.589193 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:57.589155    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f4559f84c-tdxj7_24d61891-9970-444b-9a0d-80a7e911c420/manager/0.log"
Oct 30 05:38:57.605211 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:57.604735    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f4559f84c-tdxj7_24d61891-9970-444b-9a0d-80a7e911c420/manager/0.log"
Oct 30 05:38:58.279259 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:58.277722    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-757f4bf64c-szwjz_36ea66f0-e11f-4da2-a421-1498c0287776/authorino/0.log"
Oct 30 05:38:58.707474 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:58.707394    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vmffb_93fe6d14-eca6-492a-9237-75362698993f/kuadrant-console-plugin/0.log"
Oct 30 05:38:58.911692 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:58.911647    4874 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9xfkr/perf-node-gather-daemonset-sgcql"]
Oct 30 05:38:58.917097 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:58.917062    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-sgcql"
Oct 30 05:38:58.935787 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:58.935763    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xfkr/perf-node-gather-daemonset-sgcql"]
Oct 30 05:38:58.937503 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:58.937476    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7478d1d5-5ea5-49bd-ae61-281815e49b3d-proc\") pod \"perf-node-gather-daemonset-sgcql\" (UID: \"7478d1d5-5ea5-49bd-ae61-281815e49b3d\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-sgcql"
Oct 30 05:38:58.937990 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:58.937962    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7478d1d5-5ea5-49bd-ae61-281815e49b3d-lib-modules\") pod \"perf-node-gather-daemonset-sgcql\" (UID: \"7478d1d5-5ea5-49bd-ae61-281815e49b3d\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-sgcql"
Oct 30 05:38:58.938115 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:58.938009    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7478d1d5-5ea5-49bd-ae61-281815e49b3d-sys\") pod \"perf-node-gather-daemonset-sgcql\" (UID: \"7478d1d5-5ea5-49bd-ae61-281815e49b3d\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-sgcql"
Oct 30 05:38:58.938115 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:58.938036    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmbv5\" (UniqueName: \"kubernetes.io/projected/7478d1d5-5ea5-49bd-ae61-281815e49b3d-kube-api-access-bmbv5\") pod \"perf-node-gather-daemonset-sgcql\" (UID: \"7478d1d5-5ea5-49bd-ae61-281815e49b3d\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-sgcql"
Oct 30 05:38:58.938115 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:58.938061    4874 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7478d1d5-5ea5-49bd-ae61-281815e49b3d-podres\") pod \"perf-node-gather-daemonset-sgcql\" (UID: \"7478d1d5-5ea5-49bd-ae61-281815e49b3d\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-sgcql"
Oct 30 05:38:59.039087 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:59.039041    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7478d1d5-5ea5-49bd-ae61-281815e49b3d-lib-modules\") pod \"perf-node-gather-daemonset-sgcql\" (UID: \"7478d1d5-5ea5-49bd-ae61-281815e49b3d\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-sgcql"
Oct 30 05:38:59.039315 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:59.039096    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7478d1d5-5ea5-49bd-ae61-281815e49b3d-sys\") pod \"perf-node-gather-daemonset-sgcql\" (UID: \"7478d1d5-5ea5-49bd-ae61-281815e49b3d\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-sgcql"
Oct 30 05:38:59.039315 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:59.039121    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmbv5\" (UniqueName: \"kubernetes.io/projected/7478d1d5-5ea5-49bd-ae61-281815e49b3d-kube-api-access-bmbv5\") pod \"perf-node-gather-daemonset-sgcql\" (UID: \"7478d1d5-5ea5-49bd-ae61-281815e49b3d\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-sgcql"
Oct 30 05:38:59.039315 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:59.039147    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7478d1d5-5ea5-49bd-ae61-281815e49b3d-podres\") pod \"perf-node-gather-daemonset-sgcql\" (UID: \"7478d1d5-5ea5-49bd-ae61-281815e49b3d\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-sgcql"
Oct 30 05:38:59.039315 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:59.039187    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7478d1d5-5ea5-49bd-ae61-281815e49b3d-lib-modules\") pod \"perf-node-gather-daemonset-sgcql\" (UID: \"7478d1d5-5ea5-49bd-ae61-281815e49b3d\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-sgcql"
Oct 30 05:38:59.039315 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:59.039220    4874 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7478d1d5-5ea5-49bd-ae61-281815e49b3d-proc\") pod \"perf-node-gather-daemonset-sgcql\" (UID: \"7478d1d5-5ea5-49bd-ae61-281815e49b3d\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-sgcql"
Oct 30 05:38:59.039315 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:59.039230    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7478d1d5-5ea5-49bd-ae61-281815e49b3d-sys\") pod \"perf-node-gather-daemonset-sgcql\" (UID: \"7478d1d5-5ea5-49bd-ae61-281815e49b3d\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-sgcql"
Oct 30 05:38:59.039315 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:59.039311    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7478d1d5-5ea5-49bd-ae61-281815e49b3d-proc\") pod \"perf-node-gather-daemonset-sgcql\" (UID: \"7478d1d5-5ea5-49bd-ae61-281815e49b3d\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-sgcql"
Oct 30 05:38:59.039598 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:59.039355    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7478d1d5-5ea5-49bd-ae61-281815e49b3d-podres\") pod \"perf-node-gather-daemonset-sgcql\" (UID: \"7478d1d5-5ea5-49bd-ae61-281815e49b3d\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-sgcql"
Oct 30 05:38:59.050493 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:59.050473    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-vg5td_b90621d3-240a-489c-9ed2-6c1913a13972/limitador/0.log"
Oct 30 05:38:59.062485 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:59.062457    4874 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmbv5\" (UniqueName: \"kubernetes.io/projected/7478d1d5-5ea5-49bd-ae61-281815e49b3d-kube-api-access-bmbv5\") pod \"perf-node-gather-daemonset-sgcql\" (UID: \"7478d1d5-5ea5-49bd-ae61-281815e49b3d\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-sgcql"
Oct 30 05:38:59.234943 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:59.234913    4874 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-sgcql"
Oct 30 05:38:59.553082 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:38:59.552997    4874 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xfkr/perf-node-gather-daemonset-sgcql"]
Oct 30 05:39:00.229921 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:00.229877    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-sgcql" event={"ID":"7478d1d5-5ea5-49bd-ae61-281815e49b3d","Type":"ContainerStarted","Data":"d96b17eafafe6528540627da2c9693144a44421142f211a553f199a48c841b99"}
Oct 30 05:39:00.230444 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:00.229949    4874 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-sgcql" event={"ID":"7478d1d5-5ea5-49bd-ae61-281815e49b3d","Type":"ContainerStarted","Data":"c7423ef35b6f1ee357bf1a1f51d11219556f24f17efd3461007b67785f4c56df"}
Oct 30 05:39:00.230444 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:00.230210    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-sgcql"
Oct 30 05:39:00.270759 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:00.270713    4874 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-sgcql" podStartSLOduration=2.270698824 podStartE2EDuration="2.270698824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:39:00.269566452 +0000 UTC m=+6059.681962040" watchObservedRunningTime="2026-10-30 05:39:00.270698824 +0000 UTC m=+6059.683094396"
Oct 30 05:39:00.546254 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:00.546160    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c679faf6-9b18-4a4b-9dee-d5946c10f168/alertmanager/0.log"
Oct 30 05:39:00.676739 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:00.676711    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c679faf6-9b18-4a4b-9dee-d5946c10f168/config-reloader/0.log"
Oct 30 05:39:00.814810 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:00.814741    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c679faf6-9b18-4a4b-9dee-d5946c10f168/kube-rbac-proxy-web/0.log"
Oct 30 05:39:00.946091 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:00.946060    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c679faf6-9b18-4a4b-9dee-d5946c10f168/kube-rbac-proxy/0.log"
Oct 30 05:39:01.076938 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:01.076845    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c679faf6-9b18-4a4b-9dee-d5946c10f168/kube-rbac-proxy-metric/0.log"
Oct 30 05:39:01.207229 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:01.207197    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c679faf6-9b18-4a4b-9dee-d5946c10f168/prom-label-proxy/0.log"
Oct 30 05:39:01.338306 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:01.338211    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c679faf6-9b18-4a4b-9dee-d5946c10f168/init-config-reloader/0.log"
Oct 30 05:39:02.694222 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:02.694190    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dgvb4_30e672ce-27b7-4997-a887-b834e6a6797d/node-exporter/0.log"
Oct 30 05:39:02.824643 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:02.824612    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dgvb4_30e672ce-27b7-4997-a887-b834e6a6797d/kube-rbac-proxy/0.log"
Oct 30 05:39:02.956905 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:02.956833    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dgvb4_30e672ce-27b7-4997-a887-b834e6a6797d/init-textfile/0.log"
Oct 30 05:39:02.959216 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:02.959187    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f4559f84c-tdxj7_24d61891-9970-444b-9a0d-80a7e911c420/manager/0.log"
Oct 30 05:39:02.961691 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:02.961667    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f4559f84c-tdxj7_24d61891-9970-444b-9a0d-80a7e911c420/manager/0.log"
Oct 30 05:39:06.244107 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:06.244076    4874 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-sgcql"
Oct 30 05:39:07.447014 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:07.446505    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-7pv6t_6e6ab204-9f50-4bd2-8b0e-0044278d820d/discovery/0.log"
Oct 30 05:39:07.542143 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:07.542096    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-7pv6t_6e6ab204-9f50-4bd2-8b0e-0044278d820d/discovery/0.log"
Oct 30 05:39:07.562269 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:07.561093    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-685cdf9dcb-sgtk5_38243efb-429f-450b-a62a-6b55c62330a6/kube-auth-proxy/0.log"
Oct 30 05:39:07.653269 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:07.651682    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-685cdf9dcb-sgtk5_38243efb-429f-450b-a62a-6b55c62330a6/kube-auth-proxy/0.log"
Oct 30 05:39:07.694486 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:07.694456    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-fh868_7059d28d-3eca-44fa-80c4-ac634a59aaf7/istio-proxy/0.log"
Oct 30 05:39:07.782550 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:07.782514    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-fh868_7059d28d-3eca-44fa-80c4-ac634a59aaf7/istio-proxy/0.log"
Oct 30 05:39:07.891264 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:07.891210    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-hwcsp_62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69/istio-proxy/0.log"
Oct 30 05:39:07.959727 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:07.959696    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-hz6cz_805efc98-2476-40e3-9613-69336ac75dca/istio-proxy/0.log"
Oct 30 05:39:07.981589 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:07.981514    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-hwcsp_62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69/istio-proxy/0.log"
Oct 30 05:39:08.048515 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:08.048431    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-hz6cz_805efc98-2476-40e3-9613-69336ac75dca/istio-proxy/0.log"
Oct 30 05:39:08.214074 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:08.214037    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-x8mpd_c2ea846a-5ce0-424a-bb80-ec759b592c37/istio-proxy/0.log"
Oct 30 05:39:08.324788 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:08.324715    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-x8mpd_c2ea846a-5ce0-424a-bb80-ec759b592c37/istio-proxy/0.log"
Oct 30 05:39:08.376172 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:08.374701    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-58744ff6df-48bdt_250dbba8-ecd9-409e-ae24-1eadbdb6f7a7/payload-pre-processing/0.log"
Oct 30 05:39:08.457923 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:08.457879    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-58744ff6df-48bdt_250dbba8-ecd9-409e-ae24-1eadbdb6f7a7/payload-pre-processing/0.log"
Oct 30 05:39:10.094816 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:10.093676    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-84c6c88b65-z5b2c_b188bb6b-c5d1-4b71-8b5e-0f1aadc20aff/console/0.log"
Oct 30 05:39:12.400583 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:12.400554    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-7pv6t_6e6ab204-9f50-4bd2-8b0e-0044278d820d/discovery/0.log"
Oct 30 05:39:12.509960 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:12.509910    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-685cdf9dcb-sgtk5_38243efb-429f-450b-a62a-6b55c62330a6/kube-auth-proxy/0.log"
Oct 30 05:39:12.662057 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:12.661961    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-fh868_7059d28d-3eca-44fa-80c4-ac634a59aaf7/istio-proxy/0.log"
Oct 30 05:39:12.879062 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:12.879031    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-hwcsp_62d4b8bb-4cc1-46c1-96e4-de05f5aa4a69/istio-proxy/0.log"
Oct 30 05:39:12.955781 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:12.955709    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-hz6cz_805efc98-2476-40e3-9613-69336ac75dca/istio-proxy/0.log"
Oct 30 05:39:13.277835 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:13.277804    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-x8mpd_c2ea846a-5ce0-424a-bb80-ec759b592c37/istio-proxy/0.log"
Oct 30 05:39:13.399590 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:13.399513    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-58744ff6df-48bdt_250dbba8-ecd9-409e-ae24-1eadbdb6f7a7/payload-pre-processing/0.log"
Oct 30 05:39:14.426693 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:14.426625    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mctfq_a6755b1f-578f-4e75-86df-be3db7533610/dns/0.log"
Oct 30 05:39:14.561076 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:14.561014    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mctfq_a6755b1f-578f-4e75-86df-be3db7533610/kube-rbac-proxy/0.log"
Oct 30 05:39:14.885657 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:14.885621    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-757f4bf64c-szwjz_36ea66f0-e11f-4da2-a421-1498c0287776/authorino/0.log"
Oct 30 05:39:14.992996 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:14.992963    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-757f4bf64c-szwjz_36ea66f0-e11f-4da2-a421-1498c0287776/authorino/0.log"
Oct 30 05:39:15.050370 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:15.050344    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vmffb_93fe6d14-eca6-492a-9237-75362698993f/kuadrant-console-plugin/0.log"
Oct 30 05:39:15.203611 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:15.203540    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vmffb_93fe6d14-eca6-492a-9237-75362698993f/kuadrant-console-plugin/0.log"
Oct 30 05:39:15.247806 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:15.247776    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-vg5td_b90621d3-240a-489c-9ed2-6c1913a13972/limitador/0.log"
Oct 30 05:39:15.382509 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:15.382480    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-vg5td_b90621d3-240a-489c-9ed2-6c1913a13972/limitador/0.log"
Oct 30 05:39:15.464827 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:15.464749    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8g477_ade9bd13-6259-48aa-92b2-52f875b960e8/dns-node-resolver/0.log"
Oct 30 05:39:17.049989 ip-10-0-138-160 kubenswrapper[4874]: I1030 05:39:17.049961    4874 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lrcg5_455076cd-b484-44b6-8a1b-b4c3c0dccab8/node-ca/0.log"